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(57) Two data centers (1 , 2) located in the vicinity 
are connected using a synchronous transfer copy func- 
tion, and one of the data centers is coupled with a third 
data center (3) disposed at a remote location by an 
asynchronous remote copying function. The order 
whereat a storage sub-system located in the vicinity has 
received data from a host is consistently guaranteed, 
and the third data center holds the data. Further, each 
storage sub-system includes a function whereby, during 
normal operation, data can be exchanged and the data 
update state can be obtained by the storage sub-sys- 
tems located in the two data centers that do not directly 
engage in data transmission. 
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Description 

BACKGROUND OF THE INVENTION 

FIELD OF THE INVENTION 5 

[0001] The present invention relates to a large area 
data storage system wherein an external storage device 
can quickly recover from a blockage that occurs due to 
a disaster, and in particular, to a large area data storage 
system wherein three or more external storage devices 
located at distances of one hundred to several hundred 
kms perform complementary operations. 

DESCRIPTION OF THE RELATED ART 

[0002] Disclosed in JP1 1338647, by the present in- 
ventor, is a method whereby doubling of a system or da- 
ta is performed synchronously or asynchronously. Fur- 
ther, disclosed in JP2000305856, by the present inven- 
tor, is a technique for asynchronously copying data to a 
remote area. 

[0003] As is described above, the present inventor 
has proposed asynchronous remote copy techniques 
whereby an external storage device (hereinafter re- 
ferred to as a storage sub-system), without receiving 
special control information specifying data order, re- 
ceives data from a large computer system, a server or 
a personal computer connected to a network, or another 
higher computer system (hereinafter referred to as a 
host), and employs asynchronous transmission to con- 
tinuously write data to a remotely situated second stor- 
age sub-system, while constantly maintaining the order 
of the data. 

[0004] Further, when data is to be copied using the 
synchronous transmission technique, the performance 
of the data update process between a host and a stor- 
age sub-system connected thereto interacts with the ex- 
ercise of the copy control process between the storage 
sub-system and a second storage sub-system located 
inthevicinityorinaremote area. Therefore, macroscop- 
ically, data exchanged by the two storage sub-systems 
are constantly being matched, and the order in which 
the data are written is also obtained. When an appropri- 
ate data transfer path is selected, the copy process ef- 
fected through the synchronous transfer of data can be 
performed even when the distance between the two 
storage sub-systems exceeds 100 km. 
[0005] Recently, awareness has grown of how impor- 
tant are the safe storage and the maintenance of data, 
giving rise to the expression of many demands, originat- 
ing in the data storage market, for viable disaster recov- 
ery systems. Conventional means devised to satisfy 
these demands generally provide for the synchronous 
and asynchronous transfer of data between two con- 
nected data storage points. However, further market 
sourced requests call for the inclusion of third and fourth 
data storage points (hereinafter referred to as data cent- 



ers), and for the construction of comprehensive, or near 
comprehensive, disaster recovery systems to service 
these data centers. 

[0006] The reasoning behind these requests is that so 
long as three or more data centers are established, even 
if a disaster strikes one of the data centers, the redun- 
dancy represented by the storage and maintenance of 
data at the remaining data centers will enable data to 
be recovered and will reduce the risk represented by the 
occurrence of a succeeding disaster. 
[0007] According to the conventional technique, ade- 
quate consideration is not given for a case wherein three 
or more data centers have been established and I/O da- 
ta is received from a host having a logical volume of only 
one storage sub-system, and the remote copy tech- 
nique is used for transmissions to multiple data centers. 
For example, for an event wherein a data center is dis- 
abled by a disaster, little consideration is given as to 
whether a logical volume that guarantees data ordercan 
be maintained between two or more remaining data 
centers, whether the update state can be maintained 
and non-matching data can be removed, and whether a 
system that can copy data relative to a vicinity and a 
remote area can be re-constructed. 
[0008] Since when a disaster will occur is an un- 
known, among a grouping of three or more data centers 
the order in which data is updated must be constantly 
maintained. 

[0009] Therefore, a large area data storage system 
must be constructed wherein a specific function is not 
uniquely provided for a host and a plurality of remote 
copying systems are coupled together, wherein re- 
ceived data having the same logical volume is distribut- 
ed to another storage sub-system situated at a nearby 
or a remote location, and wherein the storage sub-sys- 
tems of data centers constantly guarantee the order in 
which data received from the host are updated. 
[0010] To resolve the above problem, according to the 
invention, a large area data storage system copies data 
to another storage sub-system without providing a re- 
dundant logical volume for a storage sub-system. 
[001 1 ] Further, according to the present invention, the 
reconstruction of a large area storage system is as- 
sumed to be the recovery operation objective following 
a disaster. During normal operation, management infor- 
mation is directly exchanged by storage sub-systems 
that do not perform data transfer functions, and the data 
update state is monitored and controlled by each stor- 
age sub-system. Then, during a recovery operation (re- 
synchronization, or resync) following a disaster, only the 
difference between data stored in the storage sub-sys- 
tems transmitted immediately before the disaster oc- 
curs, and the exchange of hosts (fail over) and the con- 
tinuation of the application are performed immediately. 

<To constantly guarantee the order for updating data> 

[001 2] A supplementary explanation will now be given 
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for the time range for holding a data order. 
[001 3] The I/O data issued by the host is written to the 
storage sub-system, and the host receives a data-write- 
complete notification from the storage sub-system be- 
fore performing the next step. When the host does not 
receive a data-write-complete notification from the stor- 
age sub-system, or receives a blockage notification, the 
host does not normally issue the next I/O data. There- 
fore, the data writing order should be maintained when 
the storage sub-system performs a specific order hold- 
ing process before and after it transmits a write-end no- 
tification to the host. 

[0014] In the remote copy process performed by the 
synchronous transfer of data, the data to be transmitted 
and copied is written to a storage sub-system situated 
nearby or at a remote location (hereinafter referred to 
simply as a different location), and when a write-end no- 
tification is received from the storage sub-system at the 
different location, the write-end notification is reported 
to the host. Compared with when a remote copy process 
is not performed, remote copy time and data transfer 
time are increased, and the performance is delayed. 
When the connection distance for a remote copy proc- 
ess is extended, the processing time for the data trans- 
fer is increased, and the remote copy process causes 
the performance of the I/O process to be further deteri- 
orated. One of the methods used to resolve this problem 
is the asynchronous transfer of data. 
[0015] During the asynchronous transfer of data, up- 
on receiving I/O data from the host, the storage sub- 
system transmits data to a storage sub-system at a dif- 
ferent location, and returns a write-end notification to the 
host without waiting for the write-end notification from 
the storage sub-system at the different location. Thus, 
the transmission of data between the storage sub-sys- 
tems is not associated with the I/O process performed 
by the host, and can asynchronously be performed with 
the I/O process of the host. However, unless the data is 
written to the storage sub-system in a different location 
in the order whereat the data was received from the 
host, the data order may not be maintained by the stor- 
age sub-system at the different location, and data non- 
matching may occur between the two storage sub-sys- 
tems. The additional provision of a function that con- 
stantly guarantees the data order, is the best possible 
means by which to reduce occurrences of this problem. 
[0016] Compared with the storage sub-system that 
has received the host I/O data, the updating of data in 
the storage sub-system at a different location is gener- 
ally delayed. However, so long as the data is written to 
the storage sub-system following the order in which the 
data arrived from the host, there is no divergence in the 
data order, and the recovery from a blockage can be 
performed by a journal file system or a database recov- 
ery process. 

[0017] There is another method by which, without 
maintaining data order, the remote copying of the data 
order to a storage sub-system at a different location and 



the reflection of the data can be performed. According 
to this method, data from the host that have been re- 
ceived up to a specific time are transmitted to a different 
location and are collectively written to the storage sub- 
5 system. When the data received up to a specific time 
have been written, the data transfer process is terminat- 
ed, and thereafter, data transfer by remote copying is 
halted until collective writing is next performed, and 
while data transfer is halted, the data order and the con- 
to sistency of the I/O data received from the host is guar- 
anteed. 

[001 8] According to this method, the function for pro- 
viding the data order information is not required. A spe- 
cific amount of data to be updated is stored and is col- 
ts lectively transmitted, and when the writing of data to a 
remote side has been completed, the data matching is 
guaranteed. According to this method, however, when 
a blockage occurs during remote copying, the data is 
not updated while the data updating order on the remote 
20 side is maintained, so that all the data are lost. On ly dur- 
ing a period in which the data transfer by remote copying 
is halted can the data matching be guaranteed and be 
called adaptive. 

[0019] The technique of the present inventor of the 

25 "remote copying by the asynchronous transfer of data 
for constantly guaranteeing the data order" includes a 
feature that, before returning an end notification to the 
host, the storage sub-system performs a process for 
guaranteeing the data order. Since regardless of the 

30 overheard in the controller of the storage sub-system, 
or the delay time for the internal process, management 
is provided for the data order information for each block 
before returning the end notification to the host, the data 
order can be consistently guaranteed. 

35 [0020] Actually, the data order information is man- 
aged or controlled for each block during a time consid- 
erably shorter than the interval whereat the host issues 
the I/O. The time out (Timeout) value for the distribution 
of data to the storage sub-system at the remote location 

40 is set for at least one hour. The importance of this is that 
the remote copy technique of the present invention 
transmits data, together with order information, to a data 
block and writes the data in order in accordance with the 
order information. This is possible, so long as the order 

45 is correct, because even when between the local and 
remote systems the time lag for the updating of data is 
half a day, for example, this is much better than when, 
due to the non-matching of data, ail the updated data 
are lost. 

50 

SUMMARY OF THE INVENTION 

[0021 ] Three or more data centers are interconnected 
by a transfer path along which data can be transmitted 
55 synchronously and asynchronously, a communication 
line along which predetermined management informa- 
tion can be exchanged, and data update state manage- 
ment means. 
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[0022] Data update state management means is pro- 
vided for each storage sub-system, and in order to cope 
with the occurrence of a disaster that can not be predict- 
ed, the update state management means appropriately 
monitors the data update state of a storage sub-system s 
that is located in another data center, and transmits no- 
tification of the data update state of the storage sub-sys- 
tem to the others. 

[0023] Specifically, each of the storage sub-systems 
that do not directly engage in the transfer of data has a 10 
transfer state/bit map, and since, to ascertain how many 
times and at which location in a transfer block data has 
been updated, one storage sub-system transmits inquir- 
ies that the other storage sub-system responds to, a 
function for monitoring and managing the state of data is 
updating (remote copying) is provided. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0024] 20 

Fig. 1 is a diagram for explaining an example con- 
figuration of a large area data storage system ac- 
cording to the present invention; 
Fig. 2 is a conceptual diagram showing an example 25 
storage sub-system; 

Fig. 3 is a conceptual diagram for explaining the da- 
ta copy monitoring function in the configuration in 
Fig. 1; 

Fig. 4 is a diagram showing an example transfer 30 
state/bit map for implementing the present inven- 
tion; 

Fig. 5 is a schematic diagram for explaining the cop- 
ying control through a general synchronous transfer 
of data; 35 
Fig. 6 is a schematic diagram for explaining the 
asynchronous remote copy control; 
Fig. 7 is a diagram for explaining the recovery state 
of the entire configuration in Fig. 9 when a blockage 
or a disaster has occurred at a data center 2; 40 
Fig. 8 is a diagram for explaining the recovery state 
of the entire configuration in Fig. 1 when a blockage 
and a disaster occurs at a data center 1 ; 
Fig. 9 is a diagram for explaining another example 
configuration for the large area data storage system *s 
according to the present invention; 
Fig. 1 0 is a diagram for explaining an additional ex- 
ample configuration for a large area data storage 
system according to the present invention wherein 
data centers are located at least four points; so 
Fig. 1 1 is a conceptual diagram for explaining a data 
copy monitoring function in the overall configuration 
in Fig. 9; 

Fig. 1 2 is a conceptual diagram of a block constitut- 
ing a unit for managing data in a storage resource 5 $ 
for explaining a method for managing data match- 
ing through an asynchronous transfer of data ac- 
cording to one embodiment of the present inven- 



tion; 

Fig. 13 is a conceptual diagram showing data man- 
agement information for explaining the manage- 
ment method for data matching through an asyn- 
chronous transfer of data according to the embod- 
iment of the present invention; 
Fig. 14 is a conceptual diagram showing the trans- 
ferred data format for explaining the management 
method for data matching through an asynchronous 
transfer of data according to the embodiment of the 
present invention; 

Fig. 15 is a conceptual diagram showing data man- 
agement information, which is managed by a stor- 
age sub-system 2, for explaining the management 
method for data matching through an asynchronous 
transfer of data according to the embodiment of the 
present invention; 

Fig. 1 6A is a conceptual diagram showing a large 
area data storage system of a multi-hop type; 
Fig. 16B is a diagram showing the flow of the 
processing performed by the storage sub-system in 
Fig. 16A: 

Fig. 17A is a conceptual diagram showing a large 
area data storage system of a multi-hop type; 
Fig. 17B is a diagram showing the flow of the 
processing performed by the storage sub-system in 
Fig. 

17A: 

Fig. 1 8 is a diagram showing the state of data trans- 
mission between the storage sub-systems when the 
multi-hop type is being switched from temporary 
use to normal use; 

Fig. 1 9 is a diagram showing the state of data trans- 
mission between the storage sub-systems after the 
multi-hop type has been switched from temporary 
use to normal use; 

Fig. 20A is a conceptual diagram showing a large 
area data storage system of a multi-hop type; 
Fig. 20B is a diagram showing the flow of the 
processing performed by the storage sub-system in 
Hg. 

20A: 

Fig. 21 A is a conceptual diagram showing a large 
area data storage system of a mufti-hop type; 
Fig. 21 B is a diagram showing the flow of the 
processing performed by the storage sub-system in 
Fig. 

20A: 

Fig. 22 is a diagram showing the state of data trans- 
mission between the storage sub-systems when the 
multi-hop type is being switched from temporary 
use to normal use; 
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Fig. 23 is a diagram showing the state of data trans- 
mission between the storage sub-systems when the 
multi-hop type is being switched from temporary 
use to normal use; 

Fig. 24 is a diagram showing the state of data trans- 
mission between the storage sub-systems after the 
multi-hop type has been switched from temporary 
use to normal use; 

Figs. 25A to 25C are diagrams for explaining a var- 
iation of the blockage recovery method of a multi- 
hop type; 

Figs. 26A and 26B are diagrams for explaining an- 
other variation of the blockage recovery method of 
a multi-hop type; 

Figs. 27A and 27B are diagrams for explaining a 
further variation of the blockage recovery method 
of a multi-copy type; 

Figs. 2BA and 28B are diagrams for explaining a still 
further variation of the blockage recovery method 
of a multi-copy type; 

Fig. 29 is a conceptual diagram showing a large ar- 
ea data storage system for explaining a method for, 
when a blockage occurs, selecting a storage sub- 
system that serves as a proxy for the current oper- 
ation; and 

Fig. 30 is a diagram showing a table for managing 
the state of the transmission of data to each sec- 
ondary storage sub-system, for a method according 
to the embodiment for managing data in a cache 
memory. 

DETAILED DESCRIPTION OF THE EMBODIMENTS 

[0025] Storage sub-systems located at three or more 
data centers are interconnected by synchronous trans- 
fers of data, and by an asynchronous remote copy tech- 
nique for constantly and sequentially guaranteeing the 
order of data. Thus, a storage sub-system of a primary 
data center receives data from a host, and transmits the 
data to each of the storage sub-systems of the data 
centers at the remaining two or more points, while main- 
taining the order wherein the host updated the data. 
[0026] Since the data is thereby rendered redundant 
while maintenance of the order wherein the host updat- 
ed the data is guaranteed, even when a disaster or a 
blockage occurs at the primary data center, the storage 
sub-systems of the remaining data centers need only 
transmit the differential data among themselves, so that 
the recovery of the remote copy operation can be quickly 
effected or the data loss can be minimized. 

Synchronization and asynchronization> 

[0027] First, copying through the synchronous trans- 
fer of data or the asynchronous remote copying is de- 
fined by referring to Figs. 5 and 6. 
[0028] Du ring the copying process performed through 
the synchronous transfer of data, when a host 1 issues 



a data update (write) instruction to a storage sub-system 
1 , and when the data to be written are also those that 
are to be written to a storage sub-system 2 that is locat- 
ed in the vicinity, a data update end notification is trans- 

5 mitted to the host after the data has been updated (writ- 
ten), as instructed, relative to the storage sub-system. 
In this embodiment, the vicinity is a so-called metropol- 
itan network included within a 100 km range. 
[0029] Specifically, for the remote copying through the 

10 synchronous transfer of data (Fig. 5), an updated data 
block is received from the host 1 by the storage sub- 
system 1 (1), and is transferred from there to a storage 
sub-system 2 (2). After the data block has been written, 
a data block write-end is received by the storage sub- 
's system 1 (3), and finally, is transmitted to the host 1 (4). 
When the intermediate process fails, the occurrence of 
a writing blockage is reported to the host 1 . 
[0030] When copying through the synchronous trans- 
fer of data is performed, macroscopically the data in the 

20 near sub-system 1 connected to the host 1 constantly 
matches the data stored in the farther distant storage 
sub-system 2 located in the vicinity. Thus, even the func- 
tion of one of these storage sub-systems is lost due to 
a disaster, the complete state immediately before the 

25 disaster occurred is held by the other storage sub-sys- 
tem, and the processing can be quickly resumed by the 
remaining systems. The fact that the data are consist- 
ently matched macroscopically indicates that during the 
performance of the synchronous transfer function, the 

30 data may not be matched by the unit (usee, msec) of the 
processing time of a controller or an electric circuit, but 
at the time whereat the data updating is completed, the 
data is always matched. This is because the storage 
sub-system 1 nearer the host 1 can not complete the 

35 updating process unless the updated data is reflected 
to the storage sub-system in the vicinity. 
[0031] In the asynchronous remote copy process 
(Fig. 6), when the host 1 issues a data update (write) 
instruction to the nearest storage sub-system connect- 

40 ed thereto, and when the data to be written is also that 
which is to be written to the storage sub-system situated 
at a remote location, the end of the updating process is 
reported to the host 1 as soon as it is completed by the 
storage sub-system 1 , and the data updating (reflecting) 

45 is performed by the storage sub-system 2 at the remote 
location asynchronously with the process performed by 
the storage sub-system 1 near the host 1 . 
[0032] Thus, since the data updating is terminated 
within the processing time required by the nearer stor- 

50 age sub-system 1 , the host 1 is not kept waiting longer 
than the transfer time or the storing process time due to 
the storage of data in the storage sub-system 2 sited at 
the remote location. The remote location is a point, fur- 
ther distant than the vicinity, in a so-called transconti- 

55 nental network, wherefor data communication or trans- 
fer is enabled without any restriction on the distance. 
[0033] More specifically, in the asynchronous remote 
copying process, the updated data block is received 
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from the host 1 by the storage sub-system 1 (1 ), and the 
end of the writing of the updated data block is transmit- 
ted to the host 1 (2). Further, the storage sub-system 1 
transmits the data, in accordance with its own schedule, 
to the storage sub-system 2 asynchronously with the s 
process performed by the host 1 . 
[0034] Because of the complicated data transfer path 
to the remote location or to the vicinity and the en route 
bottleneck of the data transfer path, the order of the data 
that is being transmitted is not guaranteed (see an ellip- 
tical block indicated by a broken line in Fig. 6). 
[0035] Generally, in order to improve the data transfer 
performance, or in many cases, to increase the transfer 
speed, the data may be transmitted along multiple trans- 
fer paths by a transmission source. Further, when the 
destination is far distant, even when from the source on- 
ly one transfer path is extended outward, the route taken 
to the destination is not always a single path because 
communication relay devices, such as a switch and a 
router, are located between the source and the destina- 
tion. And when multiple paths are employed for the 
transmission of data, depending on the path taken, time 
differences may be generated since data may be trans- 
mitted along a fast path or a slow path, so that the order 
in which data arrives at the transfer destination does not 
always correspond with the order in which the data is 
transmitted by the source. 

[0036] In an example enclosed by an ellipse in Fig. 6, 
data is transmitted along the data transfer path in the 
order Data#1 , Data#2, Data#4 and Data#3, while at the 
storage sub-system 2, the data is updated in the order 
Data#1 , Data#2, Data#3 and Data#4 because the stor- 
age sub-system 2 sorts the received data and rearrang- 
es them in the correct order. Therefore, since the data 
updating order is maintained even if an unexpected dis- 
aster has occurred immediately after the update 
processing, the database and the journal file system of 
the storage sub-system 2 can be recovered. On the con- 
trary, performing the recovery process is impossible 
when a disaster occurs immediately before the updating 
process. However, when the data transmission is con- 
tinuously performed between the storage sub-systems, 
the problem presented by the non-matching of data can 
be minimized, and macroscopically, the order in which 
data is to be updated can be obtained consistently. 
[0037] In this embodiment, when the host 1 receives 
a data block and transmits it to the storage sub-system 
2, the host 1 provides for the data sequence number 
information indicating the data updating order. There- 
fore, the storage sub-system 2 can sort the data based 
on the sequence number information, guarantee the or- 
der, and complete the storing of the data. After the proc- 
ess sequence required for the data transmission is com- 
pleted, the data order is stored in the storage sub-sys- 
tem 2 situated at the remote location. As is described 
above, when the data process inherent to the asynchro- 
nous copying is continuously performed (asynchronous 
remote copying), the data updating order can be con- 



stantly guaranteed. 

[0038] The asynchronous remote copying includes as 
a feature the extension of the distance between the stor- 
age sub-systems 1 and 2 without any deterioration in 
the performance of the host 1 , and the consistent guar- 
antee of the data order. Thus, when the user of the large 
area data storage system carries out his or her job, the 
matching of the databases or the journal file systems at 
a substantially arbitrary time can be obtained by the stor- 
age sub-system situated at a remote location. 

<Large area data storage system 1> 

[0039] Fig. 1 is a diagram showing the general con- 
figuration of a large area data storage system according 
to the invention. Fig. 9 is a diagram showing the general 
configuration of another large area storage system ac- 
cording to the invention. Fig. 10 is a diagram showing 
an example application that uses a combination of the 
configurations in Figs. 1 and 9. 

[0040] In Fig. 1 , a storage sub-system is located in 
each of three data centers. Multiple storage sub-sys- 
tems may be located in each data center, or multiple 
storage sub-systems that include a remote copy func- 
tion may be connected to each data center. The appli- 
cation is executed by a host connected to the data cent- 
er 1 , and the data transfer paths between the host and 
the data center 1 are a fiber channel, a main frame in- 
terface, an ethernet LAN, a public line, and the Internet 
or another dedicated line. 

[0041] The data center 1 and the data center 2 are 
present in the vicinity, and can exchange data through 
synchronous transmission. The data center 1 and the 
data center 3 are relatively situated at remote locations, 
and can exchange data through an asynchronous re- 
mote copying technique. 

[0042] In the normal operating form, the updated data 
that the data center 1 receives from the host is stored 
in the storage sub-system of the data center and em- 
ployed. This updated data is synchronously transmitted 
to the storage sub-system of the data center situated in 
the vicinity through a fiber channel, a main frame inter- 
face, an ethernet LAN, a public line or the Internet or 
another dedicated line. That is, macroscopically, the da- 
ta centers 1 and 2 constantly maintain the performance 
of data matching between the storage sub-systems. 
[0043] In the normal operating form, the updated data 
that the data center 1 receives from the host is transmit- 
ted to the storage sub-system of the data center situated 
at a remote location, along the same dedicated line 
while using the asynchronous remote copying tech- 
nique in the same manner as the synchronous transmis- 
sion of data. It should be noted that the same line need 
not be employed for the data centers 1 and 2 and the 
data centers 1 and 3, and the data transfer paths be- 
tween them. 

[0044] There is a long distance between the data 
center 1 and the data center 3, and the non-matching of 
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the order in which the updated data arrive occurs due 
to the transfer path between the data centers 1 and 3. 
Further, differential data that becomes non-reflected da- 
ta at the transfer destination is present in the storage 
sub-system of the data center 1 at the transfer source. 
However, according to the asynchronous remote copy 
technique of the invention, since data received from the 
host is maintained in the order that is required for the 
recovery of the database and since the file system fol- 
lowing the performance of the data process inherent to 
a predetermined asynchronous transfer of data is guar- 
anteed, the order of the data for which non-matching oc- 
curs can be recovered. As a result, the order of the up- 
dated data received from the host is maintained be- 
tween the storage sub-systems of the data center 1 and 
the data center 3. 

[0045] In order to perform the recovery process, the 
communication line along which the data is transmitted 
is laid and prepared between the data center 2 and the 
data center 3, and the updated data from the host is not 
transmitted during the normal operation of the large area 
data storage system. Further, in order to cope with the 
occurrence of a disaster or a blockage at the data center 

1, in the normal operation mode, an inquiry command 
for the data transfer process status is transmitted along 
the communication line from the data center 2 to the da- 
ta center 3, or from the data center 3 to the data center 

2. The communication lines that are laid and prepared 
are a fiber channel, a main frame interface, an ethernet 
LAN, a public line and an Internet or dedicated line. 
[0046] During normal operation, to determine whether 
the updated data is received from the host by the asyn- 
chronous remote copying performed between the stor- 
age sub-systems 1 and 3, an inquiry is transmitted along 
the communication line between the data centers 2 and 
3 using a "data transfer state inquiry command" issued 
by the storage sub -system 2. 

[0047] The "data transfer state inquiry command" is 
activated in accordance with the schedule for the stor- 
age sub-system 2. At the timing whereat data is received 
from the storage sub-system 1 through synchronous 
transmission, this command may be issued or may be 
collectively issued at a predetermined time interval. The 
predetermined time interval may be, for example, 100 
msec to 500 sec, and should be appropriate so that not 
too much time is spent in the management of a transfer 
state/bit map, which will be described later, and in the 
management of the differential data. Multiple bit maps 
may be examined upon the reception of one inquiry. 
[0048] During normal operation, data is not directly 
exchanged by the storage sub-systems 2 and 3. There- 
fore, the storage sub-system 2 issues a "data transfer 
state inquiry command" to gain an understanding of the 
data updating statuses of the storage sub-systems 1 
and 3. 

[0049] When a blockage has occurred at the data 
center 1 , the host of the data center 2 is employed to 
continue the current system operation (fail over of the 



host), and the differential data between the storage sub- 
systems 2 and 3 is transmitted by the data center 2 to 
the data center 3 along the communication line that is 
prepared to perform the recovery process. The immedi- 
5 ate recovery of large area data storage system can be 
effected only by the transmission of the differential data. 
A fail over means a change from the primary system to 
the sub-system, and used also to be called a hot stand- 
by. 

w [0050] When the data center 2 thereafter performs the 
above described asynchronous remote copying for the 
data center 3 along the communication path, as the data 
center 1 has been recovered, through the synchronous 
transfer of data the recovery process is performed be- 
15 tween the data center 2 and the data center 1 so that 
the large area data storage system existing before the 
blockage occurred can be recovered. It should be noted 
that the role of the data center 1 and the role of the data 
center 2 are exchanged before and after the blockage 
20 has occurred. 

[0051 ] As is described above, the two data centers sit- 
uated in the vicinity and the two data centers situated at 
the remote locations are unified to provide a total of 
three data centers, so that a large area data storage sys- 
25 tern connected by the remote copying technique can be 
provided. With this configuration, when a medium sized 
disaster or blockage has occurred, one of the data cent- 
ers that are interconnected by the synchronous transfer 
of data can serve as a replacement for the other. Mac- 
ao roscopically, the data in the storage sub-systems of the 
two data centers are matched by the synchronous trans- 
fer of data, and the fail over can be immediately per- 
formed. 



[0052] Since the communication line between the da- 
ta centers 2 and 3 in Fig. 1 is provided for emergency 
use, when instead of this communication line the data 
transfer path between the data centers 1 and 3 is se- 
lected after the recovery from the blockage and the dis- 
aster, the large area data storage system has the con- 
figuration shown in Fig. 9 following the recovery. 
[0053] Fig. 9 is a diagram showing an example where- 
in the storage sub-systems 1 and 2 are connected by 
the synchronous transfer of data, and the storage sub- 
systems 2 and 3 are connected by asynchronous re- 
mote copying. In the large area data storage system in 
Fig. 1 , the operation is changed from the data center 1 
to the data center 2, and by using the data center 2 as 
the main operation site, the data is synchronously trans- 
mitted by the data center 2 to the data center 1 following 
the recovery from the blockage and the disaster, while 
the data is asynchronously transmitted by the data cent- 
er 1 to the data center 3. 

[0054] In Fig. 9, the storage sub-system 1, which is 
not directly concerned with the data transmission, is- 
sues the "data transfer state inquiry" command to the 



35 <Large area data storage system 2> 



40 



45 



50 



7 



13 



EP 1 283 469 A2 



14 



storage sub-system 3, and in response to this com- 
mand, the data center 3 forwards the results to the data 
center 1 . In Fig. 10, the configuration obtained by using 
both the arrangements in Figs. 1 and 9 is shown. The 
path between the storage sub-systems 3 and 5 and the 
path between the storage sub-systems 2 and 5 corre- 
spond to the paths used for issuing and responding to 
the "data transfer state inquiry" command. 
[0055] With the above described configuration of the 
large area data storage system, even when a large dis- 
aster occurs, or blockages have continuously occurred 
in the two data centers situated in the vicinity, the fail 
over to the host of the data center 3 is performed, so 
that the data being processed by the system immedi- 
ately before the disaster occurred can be continuously 
processed and the loss of data can be minimized. 
[0056] That is, when a disaster targe enough to de- 
stroy two data centers in the vicinity has occurred, the 
storage sub-system of the data center 3 or 5 situated at 
a remote location (Figs. 1 , 9 and 1 0) can become effec- 
tive. This is because the asynchronous remote copying 
has been performed while the order of the updated data 
received from the host is maintained. It should be noted 
that data non-reflected due to the disaster can not be 
recovered. 

<Configu ration of a storage sub-system > 

[0057] Figs. 1, 9 and 10 are diagrams showing the 
combination of the copying through the synchronous 
transfer of data and the asynchronous remote copying. 
Originally, the remote copying is obtained by coupling 
one logical volume with another logical volume using the 
data transfer technique. According to the invention, the 
data received for one logical volume is transferred syn- 
chronously, and is further transmitted asynchronously 
to the vicinity and to the remote location by the remote 
copy function. 

[0058] These processes are implemented by the mi- 
cro code of the controller of the storage sub-system. The 
updated data received from the host or another storage 
sub-system is temporarily stored in a cache 5 (Fig. 2). 
At this time, the data has not yet been written by the 
RAID control to the hard disk drive of the storage sub- 
system. In the cache 5, the transfer control information 
is added to the updated data, the resultant data is trans- 
mitted to another storage sub-system by the remote 
copying technique, and the remote copying configura- 
tion using multiple storage sub-systems is implemented. 
When the combination of the synchronous transfer and 
the asynchronous transfer of data is maintained, the da- 
ta centers always hold the logical volumes with which 
the database and the file system can be recovered, 
while maintaining the order for the updating of data. 
[0059] Fig. 2 is a schematic diagram showing the con- 
figuration of the storage sub-system. 
[0060] A controller 1 comprises a channel adaptor 3, 
for the exchange of data by a host and a remote copy 



destination; and a disk adaptor 9, for controlling a hard 
disk drive 7 in a disk device 2 along a disk interface (disk 
l/F) 8. 

[0061] The channel adaptor 3 and the disk adaptor 9 
s each includes a microprocessor, and are connected to 
the cache memory 5 via a data transfer bus or control 
bus 1 1 . The bus structure is only an example, and may, 
as needed be a cross-bar structure. Further, a plurality 
of controllers 1 may be provided to form a cluster struc- 
10 ture, and a third common bus may be added to connect 
the controllers 1 . 

[0062] The cache memory 5 is used to store data that 
is to be exchanged with the host or with the remote copy 
destination. The control information, the configuration 

15 management information and the transfer state/bit map 
are stored in the control memory 6. 
[0063] The remote copy function includes a transmis- 
sion function and a reception function, and in this em- 
bodiment, the channel adaptors for receiving the I/O da- 

20 ta from the host are separately mounted. The I/O data 
received from the host is temporarily stored in the cache 
5. The transfer destination for the remote copying and 
the status management/bit map, which will be described 
later, are stored as control data in the control memory 6 

25 and are controlled by the micro code. 

[0064] The data stored in the cache 5 is written by the 
disk adaptor 9 to the hard disk drive 7 under RAID con- 
trol. As a separate process, by using the micro code the 
data is transmitted to the remote copy destination that 

30 is defined in advance. 

[0065] For example, the data received from the host 
is defined as the target for the succeeding remote copy 
process, data transmission by asynchronous transfer is 
defined, and the sequence number is provided for the 

35 data in the cache 5 in the order of the reception of data. 
The sequence number is also ID information indicating 
the data updating has been performed. The data is 
transmitted with the sequence number by the remote 
copy transmission function of the channel adaptor 3. 

40 [0066] As another example, when the remote copying 
control is defined whereby the updated block received 
from the host is connected to multiple logical volumes, 
the data inside the cache memory 5 is processed for 
synchronous transfer and also for asynchronous trans- 

45 fer, and the resultant data, together with the sequence 
number, is transmitted by the channel adaptor 3 to the 
vicinity or to the remote location. 
[0067] The example in Fig. 2 implements the present 
invention, and the present invention does not depend 

50 on the hardware configuration. This is because when 
the remote copying connection can be established be- 
tween the storage sub-systems, the present invention 
can be carried out by the logical support and the micro 
code control using the micro processor. 

55 

<Transfer state/bit map> 

[0068] Fig. 4 is a diagram showing an example table 
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for the transfer state/bit map (hereinafter referred to as 
a bit map, as needed). This table is prepared inside the 
storage sub-systems located in two data centers that do 
not directly perform data transmission in order to under- 
stand the data updating state of a partner (a storage 
sub-system placed in another data center) that will be 
paired at the time of the recovery from a disaster or a 
blockage. For example, in Fig. 1 , the data centers 1 and 
3 are paired to cope with an emergency. And in the large 
area data storage system in Fig. 9, the storage sub-sys- 
tems 1 and 3 are paired, or in Fig. 10, the storage sub- 
systems 2 and 5 and the storage sub-systems 3 and 5 
are respectively paired to cope with an emergency. 
[0069] The transfer state/bit map is required for the 
paired logical volumes, and in this invention, at least two 
transfer states/bit maps can be obtained for one logical 
volume. In accordance with a pair of storage sub-sys- 
tems and the definition of an assumption by the paired 
logical volumes, each bit map is employed to manage a 
difference with the logical volume of a partner. The block 
number in the bit map corresponds to a block that is the 
minimum unit for managing the update of the logical vol- 
ume. 

[0070] The host I/O need not be the same unit as the 
block number. The unit of the host I/O is normally 512 
bytes, at the minimum, and an upper limit is also set; 
however, these are variable. The bit map is sightly small- 
er than 50 kB or around 700 kB; however, it can have 
various sizes ranging from 20 kB to 1000 kB. One bit 
map does not always correspond to one block of the 
host I/O data. 

[0071 ] When the contents of the block corresponding 
to the block number are updated, differential manage- 
ment is conducted for all the data for the pertinent block 
number, and at the time of synchronization (resync), all 
the data for the block number is transmitted. 
[0072] For each block number, the bit map is used as 
the unit for which the logical volume is updated. And 
"Update" information to be transmitted to another logical 
volume is waited for, so that only the updated block need 
be transmitted in order to reconstruct (resynchronize) 
the pair of logical volumes used for remote copy. In other 
words, when the Update flag is On (1 in the embodiment 
in Fig. 4), it means that the pertinent data is the trans- 
mission target. And once a normal Update is performed 
by the command unit of the host, the Update flag is set 
to 0, based on the counter value of 0. 
[0073] The bit map has a further counter value 
whereat updates repeated multiple times are recorded 
using the same block number. The counter value is 0 for 
no update, or is 3 when the updating was repeated three 
times. When the size of a data block represented by a 
block number is larger than a data block updated by the 
host, the counter value is employed so that only the up- 
dated data can be transmitted to the logical volume part- 
ner. 

[0074] A data copy monitoring function, which will be 
described later, compares the block number and the 



counter value that are stored in the "data transfer state 
inquiry command", which will also be described later, 
with the block number and the counter value of the bit 
map for the storage sub-system at the inquiry destina- 

5 tion. In this comparison, when the counter value stored 
in a specific storage sub-system is equal to or greater 
than the counter value included in the "data transfer 
state inquiry command", that value is transmitted to the 
specific storage su b-system and the cou nter value of the 

10 bit map of the predetermined storage sub-system is dec- 
remented by one. 

[0075] When the counter value held in the specific 
storage sub-system is smaller than the counter value 
included in the received "data transfer state inquiry corn- 
's mand", the counter value of the bit map of this storage 
sub-system is unchanged. Whether or not the counter 
value is decremented is transmitted in response to the 
"data transfer state inquiry command". 
[0076] When the counter value of the bit map of the 
20 storage sub-system is "equal to or greater than" the 
counter value included in the received "data transfer 
state inquiry command", the data updating status indi- 
cates that the data have already been stored in or written 
to the pertinent storage sub-system by the normal re- 
25 mote copying function. When the counter value of the 
bit map is "less than" the counter value included in the 
"data transfer state inquiry command", it means that da- 
ta has not yet been received. 

[0077] The counter value in Fig. 4 is finite, and when , 
30 for example, one byte is allocated as the counter value, 
the management process can not be performed more 
than 256 times. In this example, when the same block 
has been updated over 256 times, the counter value is 
not incremented any longer, and the Update flag is set 
35 permanently. That is, in Fig. 4, the information repre- 
senting "Over Flow" is stored in the counter value. 
[0078] Once this permanent setup is performed (Over 
Flow in Fig? 4), the release (entering a value of 0) of the 
Update flag of the block, which is specified in the bit map 
40 and which is permanently set, is not performed until the 
storage sub-system having this bit map acknowledges 
that the data transmission to the partner logical volume 
has been completed and the copy is established. 
[0079] The reason for the updating and the manage- 
rs ment using the counter value will now be supplementally 
explained. 

[0080] When, for example, the bit map is to be man- 
aged in correlation with a track having a data capacity 
of about 50 kB, assume that three different portions of 

so the data of 50 kB are updated at different times. The bit 
map is managed in correlation with the track because 
the recovery (re-synchronization) from a disaster or a 
blockage is performed by using the track unit. 
[0081] When the bit map is not managed by using the 

55 counter value, only the Update flag is monitored. Even 
when it is determined at a specific time that the Update 
flag is 1 , if at the following time the data is updated the 
second or the third time, the second and the following 
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data updates are missed. Since a new concept for the 
counter value is introduced and the updating of the 
same data block using the command unit received from 
the host is precisely monitored, the above described in- 
convenience can be avoided. 

[0082] An explanation will now be given for the defi- 
nition of the transfer state/bit map function implemented 
inside the controller 1 in Fig. 2 using the micro code. The 
logical volume holds the following transfer states with 
the logical volume that is paired for the remote copy. 
These states do not depend on the synchronous trans- 
fer or the asynchronous transfer. 

1) The "normal pair state" is the state wherein the 
two overlapping volumes hold the same data while 
guaranteeing the data order. 

2) The "transfer suppression bit map registration 
state" is the state wherein the data updating has not 
yet been registered in the bit map. It should be noted 
that the data has not yet been transferred to the 
paired volume. 

3) The "copy state using a bit map" means the time 
whereat the "transfer suppression bit map registra- 
tion state" is shifted to the "normal pair state". This 
state corresponds to the initial state for double cop- 
ying. 

4) The "interrupted state" is the state wherein data 
can not be transmitted due to a blockage. This state 
is registered in the bit map. 

5) The "no-pair bit map registration state" is a spe- 
cial state inherent to the present invention. This 
state occurs from the need whereat the data updat- 
ing state must be monitored and held by the two 
volumes before a disaster or a blockage occurs. 

6) The "no pair state" is the state wherein, while a 
bit map is prepared, the logical volumes have not 
yet been paired, and no information for data updat- 
ing is registered. 

[0083] The presence of the "no-pair bit map registra- 
tion state" is the feature of the present invention. As the 
proxy for this state, the suspended state, "transfer sup- 
pression bit map registration state", may be employed. 
The suspended state is the state wherein the state of 
updating data in the logical volume is managed only by 
the bit map, and the transfer using the remote copy is 
not performed. 

[0084] In this embodiment, the "no-pair bit map regis- 
tration state" is provided because the transfer state/bit 
map must be held by the pair (Fig. 3). For example, the 
following reason is applied for the large area data stor- 
age system in Fig. 1 . 

[0085] In order to monitor the data held by the data 
center 3, the data update state of the data center 3 must 
be included in the transfer state/bit map that is provided 
in accordance with the logical volume of the storage 
sub-system of the data center 2. Further, in order to 
monitor the data held by the data center 2, the data up- 
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date state of the data center 2 must be included in the 
transfer state/bit map that is provided in accordance with 
the logical volume of the storage sub-system of the data 
center 3. 

5 [0086] In the large area data storage system in Fig. 9, 
in order to cope with the occurrence of a blockage in the 
data center 2, based on the difference in the manage- 
ment information for the remote copy between the data 
centers 1 and 3, the data centers 1 and 3 must hold the 

10 "no-pair bit map registration state" in order to establish 
a pair between them. As a result, when a blockage oc- 
curs in a storage sub-system or along a data transfer 
path, the current state can be obtained, the non-trans- 
mitted data block can be stored by using the bit map, 

15 and the differential transfer of only the updated portion 
can be performed after the blockage is removed. 
[0087] The transfer state/bit map function is imple- 
mented by the micro code that carries out the above de- 
scribed control and a control table that is related to the 

20 bit map. The specific function is performed by the micro 
code, for example, of the micro processor 4 in Fig. 2 and 
the control memory 6, and as was previously described, 
this function can be freely mounted by the control pro- 
vided by the micro code. For example, the transfer state/ 

25 bit map function can be implemented by the micro proc- 
essor 1 0, or can be carried out by the controller having 
only one micro processor. 

<Operation of a large area data storage system> 

30 

[0088] Fig. 3 is a schematic diagram for explaining the 
basic control method employed when the large area da- 
ta storage system in Fig. 1 is operated normally. During 
the normal operation, the "data transfer state inquiry 

35 command" is transmitted by the storage sub-system 2 
to the storage sub-system 3. For example, upon the oc- 
currence of a blockage in the storage sub-system 1 , in 
order to transfer the actual differential data, the storage 
sub-systems 2 and 3 employ the transfer state/bit map 

40 function to perform a logical calculation for the bit maps 
of the two storage sub-systems. Based on the results, 
the storage sub-system 2 transmits only the pertinent 
data block to the storage sub-system 3. Fig. 8 is a sche- 
matic diagram showing the processing performed for re- 

45 suming the asynchronous remote copying when a 
blockage or a disaster has occurred in the data center 
1 of the large data storage system in Fig. 1 . 
[0089] In Fig. 8, during normal operation, the data is 
doubled by synchronous transmission from the data 

50 center 1 to the data center 2 nearby, and the data is cop- 
ied to the data center 3 at the remote location by i asyn- 
chronous transmission, while maintaining the data up- 
dating order. In accordance with the schedule of the 
storage sub-system 2 of the data center 2, the data 

55 transfer state inquiry command is issued to the data 
center 3, and the data centers 2 and 3 exchange the 
management information to manage the difference in 
the data. 
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[0090] When a disaster or a blockage has occurred in 
the data center 1 , the storage sub-system of the data 
center 2 transmits differential data to the data center 3 
using asynchronous transfer, and the system operation 
performed between the data center 2 and the remote 
data center 3 can be immediately recovered. 
[0091] In Fig. 3, two transfer states/bit maps are held 
by one logical volume, and each volume has functions 
that use these bit maps. Relative to the storage sub- 
systems 2 and 3, the storage sub-system 1 has a func- 
tion corresponding to transfer state/bit map #1 and a 
function corresponding to transfer state/bit map #2. 
[0092] For a synchronous transfer and an asynchro- 
nous transfer, the storage sub-systems 2 and 3 have the 
functions of transfer state/bit map #3 and #6. During nor- 
mal operation, the functions #1 and #3, and #2 and #6, 
hold the "normal pair state". 

[0093] The functions of the transfer state/bit map #4 
and #5 are provided for the storage sub-systems 2 and 
3. When the large data storage system is normally op- 
erated, the functions of transfer state/bit map #4 and #5 
hold the "no-pair bit map registration state". 
[0094] The function of transfer state/bit map #4 per- 
forms differential management relative to the logical vol- 
ume of the storage sub-system 3, and the function of 
transfer state/bit map #5 performs differential manage- 
ment relative to the logical volume of the storage sub- 
system 2. 

[0095] In a configuration extended from that in Fig. 10, 
when the controller 1 of the storage sub-system is in- 
stalled in the first data to receive the I/O from the host, 
and includes N copy destinations through synchronous 
transfer and M destinations through asynchronous re- 
mote copying, the controller 1 includes N + M transfer 
state/bit map functions. Accordingly, a corresponding 
storage sub-system (copy destination) in the vicinity or 
a remote location also includes a transfer state/bit map 
function. As a result, even when a blockage has oc- 
curred in the controller 1 or along the data transfer path, 
the current state can be obtained, non-transmitted data 
blocks can be stored using the bit map, and the differ- 
ential transmission for only the updated portion can be 
preformed when the blockage is removed. 

<Data copy monitoring function> 

[0096] The data copy monitoring function will now be 
described. This function includes a bit map control func- 
tion, a remote copy status management function, a con- 
figuration management function, a data transfer state in- 
quiry command control function, and a remote copy data 
transfer instruction function. 

[0097] The controller of the storage sub-system 2 in 
Fig. 3 receives through synchronous transfer a data 
block from the storage sub-system 1 . The data is stored 
in the cache memory of the storage sub-system 2, and 
is also recorded by the disk drive. At this time, the per- 
tinent data block is registered in the bit map in Fig. 4 by 



transfer state/bit map function #4. 
[0098] The "data transfer state inquiry command" in- 
cluding the block number and the counter value is is- 
sued to the storage sub-system 3 by the storage sub- 

5 system 2. This command may be issued based on the 
synchronous transfer of data, or in accordance with the 
unique schedule of the storage sub-system 2. 
[0099] The controller of the storage sub-system 3 re- 
ceives the "data transfer state inquiry command" from 

10 the storage sub-system 2, and extracts the block 
number and the counter value for the transfer state/bit 
map, and compares them with the block number and the 
countervalueforthe transfer state/bit map #5 of the stor- 
age sub-system 3. 

is [0100] When the block number of the transfer state/ 
bit map #5 indicates an Update flag of 1 (update), and 
the counter value is equal to or greater than the received 
counter value, it is assumed that the data concerning 
the synchronous transfer matches the data concerning 

20 the asynchronous remote copying, and the counter val- 
ue is incremented by 1 based on the corresponding 
block number of the transfer state/bit map #6. 
[0101] When the resultant counter value is "0", the 
Update flag is set to "0". And when the counter value is 

25 "Over Flow", no further process is performed. 

[0102] Furthermore, when the counter value regis- 
tered at transfer state/bit map #5 is less than the counter 
value extracted from the inquiry command received 
from the storage sub-system 2, or when the Update flag 

30 is "0" (Off) and no update is performed, the updating to 
#5 is not performed, and this state is transmitted to the 
storage sub-system 2 as the response for the data trans- 
fer state inquiry command. 

[0103] When the transfer state/bit map function #5 
35 decrements the counter value of the transfer state/bit 
map function #6, this means that the data block that has 
been transmitted by the storage sub-system 1 to the 
storage sub-system 2 using a synchronous transfer has 
also been transmitted by the storage sub-system 1 to 
40 the storage sub-system 3 using an asynchronous trans- 
fer. 

[01 04] The data copy monitoring function employs the 
response results to control the transfer state/bit map 
function of the storage sub-system 2. When the storage 

45 sub-system 3 transmits a response indicating that the 
block number and the counter value included in the "da- 
ta transfer state inquiry command" have already been 
registered (i.e., when the counter value can be decre- 
mented), similarly, the controller of the storage sub-sys- 

so tern 2 employs the transfer state/bit map function to dec- 
rement the counter value and to set the Update flag. 
[01 05] When the response to the command indicates 
that the data has not yet been registered, it is assumed 
that the asynchronous transfer by the storage sub-sys- 

55 tern 1 to the storage sub-system 3 is incomplete, and 
transfer state/bit map function #4 of the storage sub-sys- 
tem 2 holds the updated state in its own bit map. This 
state is referred to when only the updated differential 



11 



21 



EP 1 283 469 A2 



22 



portion is re-synchronized later. 
[0106] At this time, when a critical blockage has oc- 
curred in the storage sub-system 1 and when the remote 
copying configuration must be reconstructed (re-syn- 
chronized) between the storage sub-systems 2 and 3, 
only the non-transmitted data, i.e., only the differential 
data block, need be transmitted by the storage sub-sys- 
tem 2 to the storage sub-system 3 by referring to the bit 
map. As a result, a "normal pair" can be immediately 
constructed merely by the transfer of the differential da- 
ta. The function for implementing this process is called 
the "data copy monitoring function". 

<Difference management method 1 performed between 
storage sub-systems that in a normal operation do not 
directly exchange data> 

[0107] When a blockage has occurred in the storage 
sub-system 2 of the large area storage system in Fig. 9, 
assume that the system operation has recovered by per- 
forming the asynchronous remote copying between the 
storage sub-systems 1 and 2. 

[0108] The controller 1 (Fig. 2) of the storage sub-sys- 
tem 1 that receives a data updating instruction from the 
host performs the following processing before the data 
is transmitted, through synchronous copy, to the logical 
volume of the controller 1 of the storage sub-system 2. 
[01 09] The position information of a block to be trans- 
mitted is stored, as update information for the logical vol- 
ume of the storage sub-system 3, in the bit map present 
in the controller 1 of the storage sub-system 1. At this 
time, when the block already transmitted has been up- 
dated by the storage sub-system 3, the counter value of 
the bit map is incremented by one. 
[0110] When the controller 1 of the storage sub-sys- 
tem 1 has completed the synchronous transfer to the 
controller 1 of the storage sub-system 2, the controller 
of the storage sub-system 1 issues an acknowledge- 
ment command along the communication line connect- 
ing the storage sub-systems 1 and 3 in order to ask 
whether the data block has been synchronously trans- 
mitted via the controller 1 of the storage sub-system 2 
to the controller 1 of the storage sub-system 3. 
[01 1 1 ] The acknowledgement command includes, for 
the updated data received from the host, the block 
number and the counter value of the data block for the 
storage sub-system. Upon receiving the acknowledge- 
ment command, the controller 1 of the storage sub-sys- 
tem 3 determines whether the data block received along 
the controller 1 of the storage sub-system 2 matches the 
block for which the acknowledgement command inquiry 
was issued. 

[0112] The controller 1 of the storage sub-system 3 
includes not only the transfer state/bit map function rel- 
ative to the logical volume of the controller 1 of the stor- 
age sub-system 2, but also a state management/bit map 
function relative to the logical volume of the controller 1 
of the storage sub-system 1 . 



[0113] When the controller 1 of the storage sub-sys- 
tem 3 receives data from the controller 1 of the storage 
sub-system 2, the controller 1 of the storage sub-system 
3 registers the state of the controller 1 of the storage 

5 sub-system 1 in the transfer state/bit map held in the 
storage sub-system 3. This bit map includes update in- 
formation relative to the block position associated with 
the address in the logical volume, and also includes the 
counter value in order to manage the updating of the 

10 same block multiple times. 

[0114] The block number and the counter value reg- 
istered in the transfer state/bit map of the controller 1 of 
the storage sub-system 3 are compared with those in- 
cluded in the acknowledgement command issued by the 

is controller 1 of the storage sub-system 1 . When the block 
numbers and counter values are matched, or the regis- 
tered counter value is equal to or greater than the coun- 
ter value of the acknowledgement command, it is ascer- 
tained that the arrival of the data has been normally 

20 completed, and the counter value of the bit map is dec- 
remented by one using the transfer state/bit map func- 
tion. 

[0115] When the results received from the controller 

1 of the storage sub-system 3 indicate that the data 
25 block has arrived at the storage sub-system 3 via the 

storage sub-system 2, the controller 1 of the storage 
sub-system 1 , as well as the controller 1 of the storage 
sub-system 3, decrements the counter value by one us- 
ing the transfer state/bit map function. 

30 [0116] Since the bit map is monitored and managed 
in the above described manner, even when a critical 
blockage, such as a disaster, has occurred in the stor- 
age sub-system 2 and data can not be exchanged by 
neither a synchronous nor an asynchronous transfer, 

35 the asynchronous remote copy configuration can be 
constructed by the storage sub-system 1 to which the 
host issues the I/O data and the storage sub-system 3 
that stores the data contents of the storage sub-system 

2 using the asynchronous remote copying. 

40 [0117] At this time, since the transfer state/bit map 
functions of the controllers of the storage sub-systems 
1 and 3 can be employed to transmit only the differential 
data block without copying all the logical volume data, 
the asynchronous remote copying configuration can be 

45 immediately constructed. 

<Difference management method 2 performed between 
storage sub-systems that in a normal operation do not 
directly exchange data> 

50 

[01 18] In the large area data storage system in Fig. 1 , 
the transfer state/bit map function is prepared for each 
logical volume, in order to manage the data updating 
states of the paired logical volumes, i.e., the storage 
55 sub-systems 1 and 2 and the storage sub-systems 1 and 
3. 

[01 19] When a blockage has occurred in the controller 
1 of the storage sub-system 1 , and neither the copying 
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using a synchronous transfer nor the asynchronous re- 
mote copying can be continued any longer, first, the con- 
trollers 1 of the storage sub-systems 2 and 3 copy the 
differential data to match the two data sets. Then, the 
asynchronous remote copying is established between 
the storage sub-systems 2 and 3. 
[0120] The controller 1 of the storage sub-system 1 , 
which has received from the host data to be updated, 
uses a synchronous transfer to transmit a data block to 
the controller 1 of the storage sub-system 2. Upon re- 
ceiving the data block, the controller 1 of the storage 
sub-system 2 stores the position information (block 
number) of the received data block in its own transfer 
state/bit map in order to compare the received data with 
the management information for the logical volume 
dominated by the controller 1 of the storage sub-system 
3. The transfer state/bit map function increments the 
counter value by one when the received data block is 
updated, and the data block updating performed multi- 
ple times can be recorded. 

[0121] After the controller 1 of the storage sub-system 
2 has registered predetermined management informa- 
tion in the transfer state/bit map, along the data transfer 
path connecting the controller 1 of the storage sub-sys- 
tem 2 to the controller 1 of the storage sub-system 3, 
the controller 1 of the storage sub-system 2 issues, to 
the controller 1 of the storage sub-system 3, an acknowl- 
edgement command asking whether the data block has 
arrived at the storage sub-system 3. 
[0122] The acknowledgement command includes a 
block number, which is position information for a data 
block that the controller 1 of the storage sub-system 2 
has received from the storage sub-system 1 through the 
synchronous transfer, and a counter value, which indi- 
cates the times at which the data block was updated. 
[0123] The controller 1 of the storage sub-system 3 
employs its own transfer state/bit map function to store, 
in the bit map, the position information (block number) 
and the counter value of the data block that is received 
from the controller 1 of the storage sub-system 1 by us- 
ing the asynchronous remote copying technique, so that 
the block number and the counter value can be com- 
pared with the management information of the logical 
volume dominated by the controller 1 of the storage sub- 
system 2. Then, the controller 1 of the storage sub-sys- 
tem 3 compares the values in the bit map with the cor- 
responding values included in the acknowledgement 
command. 

[0124] The block number and the counter value, 
which are included in the acknowledgement command 
issued by the storage sub-system 2 to the storage sub- 
system 3, are compared with the management informa- 
tion, which the controller 1 of the storage sub-system 3 
holds for the logical volume dominated by the controller 
1 of the storage sub-system 2. When the counter value 
is equal to or greater than that included in the acknowl- 
edgement command, the counter value of the data block 
is decremented by one using the transfer state/bit map 



function. 

[01 25] When the decremented counter value reaches 
0, it is assumed that there is no differential data between 
the storage sub-systems 2 and 3, and the counter value 

5 is erased from the bit map. When the comparison results 
are not matched, the controller 1 of the storage sub-sys- 
tem 3 does not operate the counter value of the bit map. 
[0126] The controller 1 of the storage sub-system 3 
transmits the determination results to the controller 1 of 

10 the storage sub-system 2 as a response to the acknowl- 
edgement command. When the controller 1 of the stor- 
age sub-system 2 refers to these results and decre- 
ments the counter value, it is ascertained that between 
the storage sub-systems 2 and 3 the same data block 

is has been normally updated . 

[0127] When a data block to be updated is not re- 
ceived by the storage sub-system 3, it is assumed that 
the data block to be updated is stored only in the storage 
sub-system 2. The controller 1 of the storage sub-sys- 

20 tern 2 stores this data block by using its own transfer 
state/bit map function. 

[0128] When the controller 1 of the storage sub-sys- 
tem 2 receives from the controller 1 of the storage sub- 
system 3 a response relative to the acknowledgement 

25 command, and when the data block to be updated has 
not yet been transmitted to the storage sub-system 3, 
the counter value in the transfer state/bit map that is held 
by the controller 1 of the storage sub-system 2 and that 
corresponds to the updated state of the logical volume 

30 of the storage sub-system 3 is not decremented. This 
indicates that the data block for updating the bit map is 
differential data between the storage sub-systems 2 and 
3. 

[0129] When the data has arrived, the counter value 
35 of the data block for updating the transfer state/bit map 
is decremented by one. And when the counter value 
reaches 0, the storage sub-systems 2 and 3 assume 
that the data block concerning the updating is the same 
and there is no non-matching data, and do not regard 
40 the data block as the target for the copying of differential 
data. 

[0130] As is described above, during a normal oper- 
ation, since the controllers of the storage sub-systems 
that do not directly exchange data manage the differen- 
<5 tial data between the logical volumes while assuming a 
recovery from a disaster or a blockage is effected. Thus, 
the differential data need only be copied between the 
storage sub-systems, and non-matching data can be re- 
moved quickly. 

50 

<Operation of a system after fail over> 

[01 31 ] While referring to Fig. 7, a brief explanation will 
now be given for the operation when the state of the 
55 large area data storage system in Fig. 1 is shifted by fail 
overto a configuration in Fig. 9. When a critical blockage 
has occurred in the storage sub-system 1 in Fig. 3, in 
the storage sub-system 2 in Fig. 9, or in the storage sub- 
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system 1 , 2 or 4 in Fig. 1 0, as is shown in Fig. 7, recovery 
of the remote copy configuration is effected by the re- 
maining two or more storage sub-systems. 
[0132] According to the present invention, the differ- 
ential copy need only be copied between the logical vol- 
umes (the storage sub-systems 1 and 3) that do not di- 
rectly relate to the data transfer, a remote copy pair can 
be immediately generated, and the remote copy opera- 
tion can be resumed. 

[0133] If the present invention is not applied, in the 
configuration in Fig. 3 the storage sub-system 2 must 
fully copy the stored data to the storage sub-system 3, 
or in the configuration in Fig. 9 the storage sub-system 
1 must fully copy the stored data to the storage sub- 
system 3, in order to provide the remote copy configu- 
ration between the storage sub-systems 2 and 3 in Fig. 
3, or between the storage sub-systems 1 and 3 in Fig. 
9. A large data center requires an extended period of 
time for copying, and delays the resumption of the re- 
mote copying. When a blockage or a disaster again oc- 
curs at a copy source or along a data transfer path dur- 
ing a long copy operation, the data is destroyed and lost. 
[0134] The data copy monitoring function of the con- 
figuration in Fig. 9 will be briefly described while refer- 
ring to Fig. 11. 

[01 35] The data transfer state inquiry command is is- 
sued by the storage sub-system 1 to the storage sub- 
system 3. The data copy monitoring function differs par- 
tially from that in Fig. 1 . The storage sub-system 1 syn- 
chronously transmits, to the storage sub-system 1 , the 
updated data received from the host, and permits the 
storage sub-system 3 to activate the "data copy moni- 
toring function. Specifically, the storage sub-system 1 
issues the "data transfer state inquiry command", and 
employs transfer state/bit map #1 of the storage sub- 
system 1 and transfer state/bit map #3 of the storage 
sub-system 3 to register the Update flags and the coun- 
ter values and to perform a predetermined operation. 
[0136] The storage sub-system 1 issues an inquiry to 
the storage sub-system 3 to determine whether the 
same data as the data (track) the storage sub-system 1 
received from the host has been transmitted to the stor- 
age sub-system 3. When the data has not yet been re- 
ceived, the bit map for the transfer state/bit map #1 of 
the storage sub-system 1 is maintained unchanged. If 
the data has arrived, i.e., if the block number and the 
counter value of the bit map of the transfer state/bit map 
function #3 are the same, the Update flag and the bit 
map for the transfer state/bit map function #1 are delet- 
ed. 

<Other process for re-synchronization> 

[0137] When an error or a defect occurs in the re- 
sponse to the "data transfer state inquiry command" de- 
tected by the data copy monitoring function, or when a 
defect occurs in the transfer state/bit map function, the 
difference management is inhibited, which concerns the 



recovery process that should be performed upon the oc- 
currence of a blockage or a disaster. 
[0138] For the transfer state/bit map function, the bit 
map includes a storage area for an finite counter value. 

5 When the same data block is updated over the finite val- 
ue (overflow), even if the redundancy is maintained later 
by the two or more storage sub-systems, the data block 
is always regarded as the update target when the re- 
synchronization process or the difference copy process 

10 is performed after a blockage or a disaster has occurred. 
[0139] In the normal operation, when a response is 
not issued for a predetermined period of time relative to 
an inquiry (acknowledge command) that is exchanged 
among the storage sub-systems that do not directly 

15 transmit data, it is assumed that the time has expired 
and the re-synchronization process is inhibited, without 
performing the reconstruction of a pair of logical vol- 
umes using asynchronous remote copying, or the trans- 
mission of only differential data. This is because, since 

20 the data updated state of the logical volume to be paired 
can not be obtained, it is not appropriate to perform the 
reconstruction of the pair of logical volumes. 

<Management of matching of data through an 
25 asynchronous transfer> 

[0140] Assume that the storage sub-systems 1 and 2 
connected to the host are operated using asynchronous 
transfers whereby the data is copied from the storage 

30 sub-system 1 to the storage sub-system 2. In this case, 
when the data writing order for the storage sub-system 
1 differs from the data writing order for the storage sub- 
system 2, the matching of the data for the storage sub- 
systems 1 and 2 is not guaranteed. The arrangement 

35 for avoiding the non-matching of data will now be de- 
scribed. 

[0141] First, blocks of predetermined size (e.g., 16 K 
bytes) are defined in the storage area of the resource 
for each of the storage sub-systems 1 and 2, and unique 

40 block numbers are allocated to the blocks. Then, for 
each block for which the host has written data, the cor- 
relation of the block number and the sequence number 
provided in the data writing order is entered in the control 
memory 6. For example, when as is shown in Fig. 12 

45 data is written to blocks having block numbers 56 to 59, 
the data management information in Fig. 1 3 is created 
in the control memory 6. 

[01 42] For an asynchronous transfer from the storage 
sub-system 1 to the storage sub-system 2, as is shown 

so in the transfer data format in Fig. 14, the data manage- 
ment information is attached to the data to be transmit- 
ted. Then, as is shown in Fig. 15, the storage sub-sys- 
tem 2 manages, in the control memory 6, the data man- 
agement information that is received with the data. The 

55 data management information is stored in the control 
memory 6, i.e., the combination of the sequence number 
and the block ID is stored in correlation with the position 
information in the cache memory of corresponding data. 
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The storage sub-system 2 writes, to the storage re- 
source, the data that is included in the position informa- 
tion in the cache memory in the order corresponding to 
the sequential numbers in the data management infor- 
mation. 

[01 43] As is described above, the data is written to the 
storage resource of the storage sub-system 2 in the or- 
der whereat the host has written the data to the storage 
resource of the storage sub-system 1 , so that the match- 
ing of the data in the storage sub-systems 1 and 2 can 
be guaranteed. 

<Multi-hop method> 

[0144] A large area data storage system in Fig. 16A 
comprises: a storage sub-system 1 located at a site 1 ; 
a storage sub-system 2 located at a site 2 ; and a storage 
sub-system 3 located at a site 3. The storage sub-sys- 
tem 1 is connected to a host that employs this system 
1 as storage means. The storage sub-systems 1 and 3 
are interconnected by communication means. 
[0145] The storage sub-systems 1 and 2 are em- 
ployed for synchronous transfers whereby the data is 
copied from the storage sub-system 1 to the storage 
sub-system 2. Further, the storage sub-systems 2 and 
3 are employed for asynchronous transfers whereby the 
data is copied from the storage sub-system 2 to the stor- 
age sub-system 3. The remote copy method in this form 
is thereafter called a "multi-hop method". It should be 
noted that with the multi-hop method either synchronous 
transfers or asynchronous transfers are arbitrarily set for 
communication among the storage sub-systems. Fur- 
ther, another transfer method may be employed. 
[0146] While referring to Fig. 1 6B, a detailed explana- 
tion will now be given for data difference management 
using the multi-hop method. 

[0147] The storage sub-system 1 receives, from the 
host, target data to be written and a writing request 
(Write I/O) (S121). Then, the storage sub-system 1 
writes the target data in the logical volume (first storage 
resource), provides a sequence number in the order 
whereat the data writing process was performed, and 
stores the sequence number (in a predetermined table) 
in correlation with the write position information that 
specifies the storage location in the logical volume (first 
storage resource) whereat the target data is written 
(S122). It should be noted that the write position infor- 
mation is represented using a sector number or a track 
number. 

[0148] The storage sub-system 1 transmits, to the 
storage sub-system 2 , the target data and the sequence 
number provided (S123). The transmission of the data 
and the sequence number is performed between the 
storage sub-systems after the data transmission com- 
mand has been issued, and as needed, the data write 
position information is provided for the data transmis- 
sion command. 

[0149] The storage sub-system 2 receives, from the 



storage sub-system 1 , the target data to be written and 
the sequence number, and writes them to its own logical 
volume (second storage resource). When the writing is 
completed, the storage sub-system 2 transmits a com- 

5 plete notification to the storage sub-system 1 . 

[01 50] The storage sub-system 2 transmits the target 
data and the sequence number to the storage sub-sys- 
tem 3 at an appropriate timing (S124). (In Fig. 16b, in 
order to express a time lag, the sequence number of the 

10 data transmitted by the storage sub-system 1 to the stor- 
age sub-system 2 differs from the sequence number of 
the data transmitted by the storage sub-system 2 to the 
storage sub-system 3). 

[0151] The storage sub-system 3 receives the data 

'5 and the sequence number, and transmits, to the storage 
sub-system 1 , the sequence number that is issued in 
correlation with the target data to be written (S1 25). The 
storage sub-system 1 receives the sequence number 
from the storage sub-system 3. 

20 [0152] The storage sub-system 1 examines the re- 
ceived sequence number and the correlation (table) be- 
tween the stored sequence number and the correspond- 
ing write position information. Thus, the data not reflect- 
ed to the logical volume (third storage resource) in the 

25 storage sub-system 3, i.e., the differential data, can be 
obtained. The examination is performed by deleting, 
from the table, the write position information and the se- 
quence numbers up to the write complete position that 
is received from the storage sub-system 3 (S126). 

30 [01 53] An explanation will now be given for the recov- 
ery process when the storage sub-system 2 is hatted 
due to a disaster. 

[0154] As is shown in Fig. 17A, the storage sub-sys- 
tem 1 employs, for example, a disaster detection func- 
35 tion, such as a function for monitoring a heart beat mes- 
sage, to monitor the operating state of the storage sub- 
system 2 in real time. While referring to Figs. 17A and 
17B, an explanation will now be given for the processing 
wherein, when the storage sub-system 1 detects, due 
40 to the interruption of a heart beat message, that a block- 
age has occurred in the storage sub-system 2, the con- 
tents of the storage sub-system l"and the contents of 
the storage sub-system 2 are matched by copying only 
the differential data, and the operating state of the stor- 
45 age sub-systems 1 and 3 is shifted to the temporary op- 
erating state using asynchronous transfers. 
[0155] When the storage sub-system 1 detects the 
occurrence of a blockage in the storage sub-system 2 
(S131), first, the storage sub-system 1 generates a bit 
50 map in correlation with the data storage location for a 
predetermined block unit in the logical volume (first stor- 
age resource) of the system 1 . Then, based on the cor- 
relation between the sequence number and the write lo- 
cation information, both of which are stored in the stor- 
es age sub-system 1 as is the differential data that is not 
reflected to the storage sub-system 3, the storage sub- 
system 1 renders ON a bit at the location corresponding 
to the bit map for which the data is updated (S132). 
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[0156] Then, the differential data that is stored at the 
ON location in the bit map of the logical volume of the 
storage sub-system 1 is copied from the storage sub- 
system 1 to the corresponding storage location in the 
storage sub-system 3 (S 1 33). When the copying is com- 
pleted, the temporary operation is initiated in the form 
of copying the differential data from the storage sub-sys- 
tem 2 using asynchronous transfers (S134). 
[0157] To change the operation to the temporary op- 
eration, when a blockage has occurred in the storage 
sub-system 2, not all the data need be copied from the 
storage sub-system 1 to the storage sub-system 3, and 
only the differential data need be copied. Therefore, 
when a satisfactory amount of data is not transmitted 
along the communication line between the storage sub- 
systems 1 and 3, the data stored in the logical volumes 
of the storage sub-systems can be easily synchronized. 
[01 58] Now, an explanation will be given for the proc- 
ess sequence performed when the storage sub-system 
2 is recovered and the temporary operation is changed 
to the normal operation. 

[0159] First, the storage sub-system 1 copies, to the 
logical volume (second storage resource) of the storage 
sub-system 2, all the data stored in the logical volume 
(first storage resource) of the storage sub-system 1 , and 
initiates the operation using synchronous transfers 
whereby data is copied from the storage sub-system 1 
to the storage sub-system 2. Specifically, when data is 
written to the logical volume (first storage resource) up- 
on receiving an instruction from the host, the storage 
sub-system 1 transmits the written data and the se- 
quence number to the storage sub-system 2. 
[01 60] The storage sub-system 2 writes, to the logical 
volume thereof (second storage resource), the data and 
the sequence number that are received from the storage 
sub-system 1 . When the writing process is completed, 
the storage sub-system 2 stores (in a predetermined ta- 
ble) the write location information, which specifies the 
location in the logical volume (second storage resource) 
wherein data has been written, together with the se- 
quence number provided in the data writing order. The 
data transfer state at this time is shown in Fig. 18. 
[01 61 ] Next, when the storage sub-system 3 receives 
the data and the sequence number from the storage 
sub-system 1 , the storage sub-system 3 stores the data 
in the logical volume thereof (third storage resource) 
(Fig. 18), and transmits the correlated sequence 
number to the storage sub-system 2 (not shown). 
[0162] The storage sub-system 2 receives the se- 
quence number from the storage sub-system 3. At this 
time, the storage sub-system 2 examines the received 
sequence number and the correlation between the 
stored sequence number and the corresponding write 
position information, so that data not reflected to the log- 
ical volume of the storage sub-system 3, i.e., the differ- 
ential data, can be obtained. 

[0163] Then, in the temporary operation, the asyn- 
chronous transfer process for copying the data from the 



storage sub-system 1 to the storage sub-system 32 is 
halted. After this process is halted, the storage sub-sys- 
tem 2 generates, in the control memory thereof, a bit 
map that corresponds to the data storage location for a 
s predetermined block unit of the logical volume (second 
storage resource). Then, based on the correlation 
stored in the storage sub-system 2 between the write 
position information and the sequence number for the 
differential data that is not reflected to the storage sub- 
system 3, the storage sub-system 2 renders ON a bit at 
the pertinent location of the bit map for which the data 
has been updated. 

[0164] In addition, the storage sub-system 2 trans- 
mits, to the storage sub-system 3, the differential data, 
which is not reflected to the logical volume (third storage 
resource) of the storage sub-system 3, and the write po- 
sition information, both of which are obtained from the 
bit map. 

[0165] The storage sub-system 3 receives the differ- 
ential data and the write position information, and writes 
the differential data to the data storage location that is 
designated in the logical volume (third storage resource) 
by using the write position information. Thus, synchro- 
nization can be obtained between the contents of the 
logical volume (second storage resource) of the storage 
sub-system 2 and the contents of the logical volume 
(third storage resource) of the storage sub-system. After 
the above described process is terminated, the asyn- 
chronous transfer operation is resumed by the storage 
sub-systems 2 and 3 in the normal state in Fig. 1 9. 
[0166] The shifting from the temporary operation to 
the normal operation is completed in this manner. 

<Multi-copy method> 

[0167] A large area data storage system in Fig. 20 
comprises: a storage sub-system 1 located at a site 1 ; 
a storage sub-system 2 located at a site 2; and a storage 
sub-system 3 located at a site 3. The storage sub-sys- 
tem 2 is connected to a host that employs the storage 
sub-system 2 as storage means. The storage sub-sys- 
tems 1 and 3 are interconnected by communication 
means. 

[01 68] The storage sub-systems 1 and 2 are operated 
using synchronous transfers during which the data is 
copied from the storage sub-system 2 to the storage 
sub-system 1 . The storage sub-systems 2 and 3 are op- 
erated using asynchronous transfers during which the 
data is copied from the storage sub-system 2 to the stor- 
age sub-system 3. Hereinafter, the remote copy method 
having this form is called a "multi-copy" method. It 
should be noted that either synchronous transfers or 
asynchronous transfers are arbitrarily set for the com- 
munication among the storage sub-systems when the 
multi-copy method is used. A transfer method other than 
the synchronous and the asynchronous transfer meth- 
ods may be employed. 

[0169] The data difference management method of 
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the embodiment will now be described while referring to 
Fig. 20. The storage sub-system 2 receives, from the 
host, target data to be written and a write request (Write 
I/O) (S1 61), and writes the target data to the logical vol- 
ume thereof (second storage resource). Further, the 5 
storage sub-system 2 transmits, to the storage sub-sys- 
tem 1 , the written data and the sequence number that 
is provided in the order wherein the data are written 
(S162). At the same time, the written data and the se- 
quence number that are provided are also transmitted 
to the storage sub-system 3 (S164). As well as for the 
multi-hop method, the transmission of the data and the 
sequence number between the storage sub-systems is 
performed, for example, after the data transmission 
command has been transmitted. Further, the previously 
described data write position information is attached, as 
needed, to the command. 

[01701 Tne storage sub-system 1 receives the target 
data and the sequence number from the storage sub- 
system 2, and writes the target data to the logical volume 
thereof (first storage resource). At this time, the se- 
quence number is stored (in a predetermined table) in 
correlation with the write position information that spec- 
ifies the storage location in the logical volume (first stor- 
age resource) in which the data has been written (S1 63). 
The write position information is represented using, for 
example, a sector number or a track number. 
[0171] Next, the storage sub-system 3 receives the 
target data and the sequence number from the storage 
sub-system 2, and writes the target data to the logical 
volume thereof (third storage resource). When the writ- 
ing is completed, the storage sub-system 3 transmits, 
to the storage sub-system 1 , the target data to be written 
and the sequence number that is paired with this data 
(S1 65). Thus, the storage sub-system 1 receives the se- 
quence number from the storage sub-system 3. 
[0172] The storage sub-system 1 examines the re- 
ceived sequence and the correlation of the stored se- 
quence number, and the corresponding write position 
information, so that the data not reflected to the logical 
volume (third storage resource) of the storage sub-sys- 
tem 3, i.e., the differential data, can be obtained. This 
examination is performed, for example, by deleting from 
the table the sequence numbers up to the write-end po- 
sition and the write position information that are received 
from the storage sub-system 3 (S166). 
[0173] The normal operation using the multi-copy 
method is performed in the above described manner. 
[0174] An explanation will now be given for the recov- 
ery process performed when the storage sub-system 2 
is halted due to a disaster. 

[0175] As is shown in Fig. 21 A, the storage sub-sys- 
tem 1 employs a blockage detection function, such as 
a heart beat message monitoring function, to monitor 
the operating state of the storage sub-system 2 in real 
time. An explanation will now be given, while referring 
to Fig. 21 B, for the process wherein, when the storage 
sub-system 1 detects the occurrence of a blockage in 



the storage sub-system 2 due to the interruption of the 
heart beat message, instead of the host connected to 
the storage sub-system 2, the contents of the storage 
sub-system 1 are matched with the contents of the stor- 
age sub-system 3 merely by copying the differential da- 
ta, and the operating mode for the storage sub-systems 

1 and 3 is changed to the temporary operation using 
asynchronous transfers. 

[0176] When the storage sub-system 1 has detected 
the occurrence of a blockage in the storage sub-system 

2 (S171), upon, for example, an operator's instruction, 
the operation performed by the host connected to the 
storage sub-system 2 is transferred to the sub-host con- 
nected to the storage sub-system 1 . 

[0177] Then, the storage sub-system 1 generates, in 
the control memory 6, a bit map that corresponds to the 
data storage location for a predetermined block unit of 
the logical volume (first storage resource) for the stor- 
age sub-system 1 . And, based on the correlation be- 
tween the sequence number and the updated data po- 
sition information, both of which are stored in the storage 
sub-system 1 as differential data that is not reflected to 
the storage sub-system 3, the storage sub-system 1 
renders ON the bit at the pertinent position of the bit map 
for which the data has been updated (S172). 
[0178] Further, the differential data, which is stored in 
the logical volume of the storage sub-system 1 at the 
position corresponding to the position in the bit map 
where the bit has been rendered ON, is copied from the 
storage sub-system 1 to the storage sub-system 3 
(S173). When the copying is completed, the temporary 
operation is initiated in the form where the data is copied 
from the storage sub-system 1 using a synchronous 
transfer (S174). 

[0179] To change to the temporary operation, even 
when a blockage has occurred in the storage sub-sys- 
tem 2, not all the data in the storage sub-system 1 need 
be copied to the storage sub-system 3, only the differ- 
ential data. Therefore, even when a satisfactory amount 
of data is not transmitted along the communication line 
between the storage sub-systems 1 and 3, the data 
stored in the logical volumes of the storage sub-systems 
can be easily synchronized. 

[0180] An explanation will now be given for the proc- 
ess sequence performed when the storage sub-system 
2 is recovered from the blockage and the temporary op- 
eration is changed to the normal operation. 
[0181] First, the storage sub-system 1 copies all the 
data stored in its logical volume (first storage resource) 
to the logical volume (second storage resource) of the 
storage sub-system 2, and the operation is initiated us- 
ing synchronous transfers wherein data is copied from 
the storage sub-system 1 to the storage sub-system 2. 
At this time, the asynchronous transfers between the 
storage sub -systems 1 and 3 are also continued. The 
storage sub-system 1 transmits, to the storage sub-sys- 
tem 2, the data written by the host and the sequence 
number provided in the data writing order. The storage 
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sub-system 1 also transmits to the storage sub-system 
3 the written data and the sequence number that were 
provided. The storage sub-system 2 stores the correla- 
tion between the write position information, which spec- 
ifies the position of its logical volume (second storage s 
resource) whereat the data was written, and the se- 
quence number, which is provided in the data writing or- 
der (prepares a position information management ta- 
ble). The operating state at this time is shown in Fig. 22. 
[0182] The storage sub-system 3 receives the data 
and the sequence number from the storage sub-system 
1 , stores the data in its own logical volume (third storage 
resource), and transmits the correlated sequence 
number to the storage sub-system 2. 
[0183] The storage sub-system 2 receives the se- 
quence number from the storage sub-system 3. The 
storage sub-system 2 then compares the received se- 
quence number with the con-elation stored in the storage 
sub-system 2, so that the data not reflected to the logical 
volume of the storage sub-system 3, i.e., the differential 
data, can be obtained. 

[0184] Then, during the temporary operation, the 
asynchronous transfer copying of the data from the stor- 
age sub-system 1 to the storage sub-system 3 is halted. 
After the asynchronous transfer is halted, the storage 
sub-system 2 generates, in its control memory, a bit map 
that is correlated with the data storage position for a pre- 
determined block unit of the logical volume (second stor- 
age resource) of the storage sub-system 2. Then, based 
on the correlation between the sequence number and 
the write position information that are stored in the stor- 
age sub-system 2 for the differential data that is not re- 
flected to the storage sub-system 3, the storage sub- 
system 2 renders ON a bit at the pertinent position in 
the bit map for which the data has been updated. 
[01 85] Next, when the storage sub-system 2 obtains, 
from the bit map, the differential data that is not yet re- 
flected to the logical volume (third storage resource) of 
the storage sub-system 3 and the write position infor- 
mation, the storage sub-system 2 transmits them to the 
storage sub-system 2. 

[0186] The storage sub-system 3 receives the differ- 
ential data and the write position information, and stores 
the differential data in its logical volume (third storage 
resource) based on the write position information. As a 
result, synchronization can be obtained between the 
contents of the logical volume (second storage re- 
source) of the storage sub-system 2 and the contents 
of the logical volume (third storage resource) of the stor- 
age sub-system 3. The asynchronous transfer from the 
storage sub-system 2 to the storage sub-system 3 is 
then begun. The operation state at this time is shown in 
Fig. 23. 

[01 87] When the data has been written from the host 
to the storage sub-system 1 connected thereto, and 
when synchronization is obtained between the storage 
sub-systems 1 and 2, the copying of data from the stor- 
age sub-system 1 to the storage sub-system 2 is 



changed to the copying of data from the storage sub- 
system 2 to the storage sub-system 1 . That is, since the 
operation is switched while the data are synchronized, 
an extra process, such as the copying of differential da- 
ta, is not required. 

[0188] Following this, the job performed by the host 
connected to the storage sub-system 1 is transferred by 
the host connected to the storage sub-system 2. When 
the synchronous transfer copying of data from the stor- 
age sub-system 2 to the storage sub-system 3 is begun, 
the operation in the normal state in Fig. 24 is resumed. 
[0189] Through the above processing, the switching 
from the temporary operation to the normal operation is 
completed. 

<Another blockage removal method> 

[01 90] A variation of the blockage removal method will 
now be explained. 

[0191] When the storage sub-system 1 breaks down 
in the multi-hop system shown in Fig. 25 (Fig. 25A), the 
sub- host is connected to the storage sub-system 2, and 
transfers the job of the host connected to the storage 
sub-system 1 . It should be noted that the operation us- 
ing the asynchronous transfer is performed between the 
storage sub-systems 2 and 3 (Fig. 25B). 
[0192] When the storage sub-system 1 is recovered, 
first, all the data in the storage sub-system 2 is copied 
to the storage sub-system 1 , and the job of the sub-host 
is transferred by the host connected to the storage sub- 
system 1. In the above described manner, the data 
transfer direction is reversed between the storage sub- 
systems 1 and 2, and the normal operation is resumed 
(Fig. 25C). 

[0193] When a blockage has occurred in the storage 
sub-system 3 in the multi-hop system in Fig. 26 (Fig. 
26 A), the storage sub-system 3 is recovered, all the data 
is copied from the storage sub-system 2 to the storage 
sub-system 3 to obtain synchronization between the da- 
ta in the storage sub-systems 2 and 3, and the normal 
operation is resumed by performing the synchronous 
transfer copying of data from the storage sub-system 1 
to the storage sub-system 2, and by the asynchronous 
transfer copying of data from the storage sub-system 2 
to the storage sub-system 3 (Fig. 26B). 
[0194] When a blockage has occurred in the storage 
sub-system 1 in the multi-copy system in Fig. 27 (Fig. 
27 A), the storage sub-system 1 is recovered, all the data 
is copied from the storage sub-system 1 to the storage 
sub-system 1 to obtain synchronization between the da- 
ta in storage sub-systems 1 and 2, and the normal op- 
eration is resumed by performing synchronous transfer 
copying of data from the storage sub-system 2 to the 
storage sub-system 1 and by performing asynchronous 
transfer copying of data from the storage sub-system 2 
to the storage sub-system 3 (Fig. 27B). 
[0195] When a blockage has occurred in the storage 
sub-system 3 in the multi-copy system in Fig. 28 (Fig. 
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28A), the storage sub-system 3 is recovered, and all the 
data is copied from the storage sub-system 2 to the stor- 
age sub-system 3 to obtain synchronization between 
the data in the storage sub-systems 2 and 3, and the 
normal operation is resumed by performing the synchro- 
nous transfer copying of data from the storage sub-sys- 
tem 2 to the storage sub-system 1 and by performing 
the asynchronous transfer copying of data from the stor- 
age sub-system 2 to the storage sub-system 3 (Fig. 
28B). 

<Management of write position information at a copy 
source and a copy destination> 

[0196] For the transmission of data among the stor- 
age sub-systems, the data transmission source and 
destination and the use of the synchronous transfer or 
the asynchronous transfer method is designated in var- 
ious forms depending on the system configuration; for 
example, for this designation an operator may manipu- 
late each storage sub-system (in this case, when a spe- 
cific storage sub-system can not be used due to a block- 
age, a storage sub-system, as the next data transmis- 
sion source, and a storage sub-system, as the next 
transmission destination, are registered in advance 
when the system is arranged), or a system attached to 
a storage sub-system may automatically perform the 
designation. 

[0197] The correlation between the sequence number 
and the write position information is managed at the time 
whereat, for example, an operator begins to register the 
transmission source and the transmission destination 
for the storage sub-system. 

<Method for selecting a storage sub-system> 

[0198] A large area data storage system in Fig. 29 
comprises: a storage sub-system 1 ; a host 1 h connect- 
ed thereto; and storage sub-systems 2 and 3 for asyn- 
chronously receiving data from the storage sub-system 
1 . When a blockage has occurred in the host 1 h, or the 
storage sub-system 1 , one of the storage sub-systems 
2 and 3 is immediately selected as the primary storage 
sub-system, and in order to maintain reliability and se- 
curity, the data is doubly managed by the storage sub- 
systems 1 and 3. An explanation will now be given for 
the processing performed when a blockage has oc- 
curred in the host 1 h or the storage sub-system 1 . 
[0199] The storage sub-system 2 detects the occur- 
rence of a blockage in the host 1 h or the storage sub- 
system by determining, for example, whether data has 
been transmitted by the storage sub-system 1 , or by 
monitoring a heart beat message transmitted by the 
storage sub-system 1 at a predetermined time. 
[0200] Upon the detection of the blockage, the stor- 
age sub-system 2 quickly determines the primary sub- 
system, and changes the operation to the temporary op- 
eration using a sub-host 2 or 3. The selection of the pri- 



mary storage sub-system is performed as follows. First, 
upon the detection of the blockage, the storage sub-sys- 
tem 2 transmits, to the storage sub-system 3, a mes- 
sage requesting the transmission of the latest sequence 
5 number. Upon receiving this message, the storage sub- 
system 3 transmits the latest stored sequence number 
to the storage sub-system 2. 

[0201] The storage sub-system 2 compares the se- 
quence number received from the storage sub-system 

10 3 with the latest sequence number stored in the storage 
sub-system 2. The storage sub-system 2 then selects, 
as the primary storage sub-system, a storage sub-sys- 
tem that has received the later sequence number, stores 
the identifier of the selected storage sub-system as a 

is selection choice, and transmits the identifier to the stor- 
age sub-system 3. Based on the received identifier, the 
storage sub-system 3 identifies the storage sub-system 
that has been selected as the primary storage sub-sys- 
tem. 

20 [0202] During this selection process, due to matters 
such as the properties of a communication method used 
by the storage sub-systems, of the sequence numbers 
stored in the storage sub-system 2 or 3 a sequence 
number may be omitted, in this case, the latest se- 

25 quence number of the available sequential sequence 
numbers is employed for the above comparison. 
[0203] When the primary storage sub-system is se- 
lected, the matching of the data contents stored in the 
storage sub-systems 2 and 3 is obtained in order to per- 

30 form the double management of the data using the stor- 
age sub-systems 2 and 3. This matching is performed 
by copying all of the data or differential data between 
the storage sub-systems 2 and 3. When between the 
storage sub-systems 2 and 3 the data match, the stor- 

35 age sub-system selected as the primary storage sub- 
system transmits to the sub-host connected thereto a 
message indicating that the pertinent storage sub-sys- 
tem is serving as the primary storage sub-system. Upon 
receiving this message, the sub-host begins the opera- 

40 tion as a proxy. Further, double data management using 
either synchronous transfers or asynchronous transfers 
is initiated by the storage sub-systems 2 and 3. 
[0204] In the above explanation, the storage sub-sys- 
tem 2 obtains the latest sequence number from the stor- 

45 age sub-system 3 and selects the primary storage sub- 
system. However, the storage sub-system 3 may per- 
form this process. 

[0205] In addition, for a large area data storage sys- 
tem constituted by three storage sub-systems 1 to 3, an 

so example method has been explained for selecting a 
specific storage sub-system that is employed as a proxy 
when a blockage has occurred in the storage sub-sys- 
tem 1 . This method can be employed for a large area 
data storage system constituted by four or more storage 

55 sub-systems. 
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<Management of data in a cache memory> 

[0206] For a system wherein at least one secondary 
storage sub-system, which is a destination for the re- 
mote copying of data in the primary storage sub-system 
connected to a host, is connected to the primary storage 
sub-system, an example for the management of data in 
the cache memory of the primary storage sub-system 
will now be explained. 

[0207] In this system, data that do not need to be cop- 
ied (remote copying) from the primary storage sub-sys- 
tem to the secondary storage sub-system may be de- 
leted from the cache memory of the primary storage 
sub-system after the data have been written to the stor- 
age resource of the primary storage sub-system. When 
the data is to be copied to the secondary storage sub- 
system, this data must be maintained in the cache mem- 
ory at least until the data has been transmitted to the 
secondary storage sub-system. Further, when a plural- 
ity of secondary sub-systems are present as transmis- 
sion destinations, generally, the data is not transmitted 
at the same time to these secondary storage sub-sys- 
tems because of differences in communication means 
and in operations. Therefore, in this case, the data must 
be maintained until the data has been transmitted to ail 
the secondary sub-systems. 

[0208] Thus, the primary storage sub-system manag- 
es the data to determine whether the data stored in its 
cache memory has been transmitted to all the second- 
ary storage sub-systems connected to the primary stor- 
age sub-system. Specifically, for example, as is shown 
in Fig. 30, for each of the storage blocks (#1 to #n) de- 
fined in the cache memory, the primary storage sub-sys- 
tem manages a table indicating whether the data stored 
in the storage block has been transmitted to each sec- 
ondary storage sub-system. 

[0209] In this table, bit "0" indicates that the transmis- 
sion is completed, and bit "1" indicates that the trans- 
mission is incomplete. When the data from the host is 
written to the primary storage sub-system, "1 ■ is set for 
the bit that corresponds to a secondary storage sub-sys- 
tem that is defined as a transmission destination for the 
storage block to which the data is written. Among the 
M 1 " bits for a specific block, a bit for the secondary stor- 
age sub-system for which the data transmission has 
been completed is set to "0". 

[021 0] The data stored in the storage blocks, the bits 
for which have been set to "0" for all the secondary stor- 
age sub-systems, can be deleted from the cache mem- 
ory. 

[0211] In the large area data storage system in Figs. 
1 , 9 and 1 0 having three or more sites, macroscopically, 
a logical volume that can consistently guarantee the da- 
ta order whenever a disaster or a blockage occurs can 
be maintained for an arbitrary site. 
[0212] In accordance with the effects of the invention, 
when only the differential data is copied between the 
logical volumes that do not directly relate to the data 



transmission, e.g., the storage sub-systems 1 and 3 in 
Fig. 7, the pair of logical volumes for asynchronous re- 
mote copying can be generated immediately, and the 
operation of the large data storage system can be quick- 

5 |y resumed. 

[0213] Further, in the invention, since a redundant log- 
ical volume is not required in the storage sub-system in 
order to perform remote copying, the efficiency in the 
use of the memory resources of the storage sub-system 

io can be increased, and the cost performance of the stor- 
age sub-system can be improved. 
[0214] It should be further understood by those skilled 
in the art that the foregoing description has been made 
on embodiments of the invention and that various 

15 changes and modifications may be made in the inven- 
tion without departing from the spirit of the invention and 
the scope of the appended claims. 



20 Claims 

1. A remote copy control method, for a storage sub- 
system (1 ; Fig. 3) which includes a control memory 
(6; Fig. 2) for storing control data, a cache memory 

25 (5) for temporarily storing data, a micro processor 
(10) for controlling said control memory and said 
cache memory, N transmission destinations for syn- 
chronously receiving said data from said micro 
processor, and M transmission destinations for 

30 asynchronously receiving said data from said mi- 
croprocessor, said remote copy control method 
comprising: 

a first step of storing, in said control memory, 
35 transfer states/bit maps (#1 to #6) correspond- 

ing to said N + M data destinations; 
a second step of another storage sub-system 
(2; Fig. 3), which includes a transfer state/bit 
map that corresponds to one of said transfer 
40 states/bit maps, issuing to a storage sub-sys- 

tem that does not directly perform data trans- 
mission a command for transmitting an inquiry 
for a data update process state; and 
a third step of, upon receiving a response to 
45 said command at said second step, updating 

said transfer states/bit maps. 

2. A remote copy control method according to claim 1 , 
wherein the updating of said transfer states/bit 

50 maps at said third step includes the updating of a 
counter value (Fig. 4) that is included in said com- 
mand for transmitting an inquiry for said data update 
process state, and that is used to count the times 
data in one part of a data block are updated. 

55 

3. A storage sub-system (2; Fig. 3), which includes a 
control memory (6; Fig. 2) for storing control data, 
a cache memory (5) for temporarily storing data, a 



20 



39 



EP 1 283 469 A2 



40 



micro processor (10) for controlling said control 
memory and said cache memory, a transmission 
destination for synchronously receiving said data 
from said micro processor and a transmission des- 
tination for asynchronously receiving said data from 5 
said microprocessor, comprising: 

a function for issuing an inquiry for a data up- 
date process state of another storage sub-sys- 
tem (2; Fig. 3) that does not directly relate to 10 
data transmission. 

4. A storage sub-system according to claim 3, wherein 
one transfer state/bit map set (#4) is held by said 
storage sub-system and another storage sub-sys- is 
tern that does not directly perform data transmis- 
sion, and wherein said transfer bit/bit map set (#4) 

is updated by said function for transmitting an in- 
quiry for a data update process state. 

20 

5. A storage sub-system according to claim 4, wherein 
said transfer state/bit map set has an area for hold- 
ing a counter value (Fig. 4) that represents the times 
data in one part of a data block is updated. 

25 

6. A large area data storage system (Fig. 1) compris- 
ing a first storage sub-system located in a first data 
center (1 ; Fig. 1), a second storage sub-system lo- 
cated in a second data center (2) and a third sub- 
system located in a third data center (3), so 

wherein data is synchronously transmitted 
between said first storage sub-system and said sec- 
ond storage sub-system; 

wherein data is asynchronously transmitted 
between said first storage sub-system and said 35 
third storage sub-system, and said third storage 
sub-system consistently maintains the order of said 
data that has been transmitted. 

7. A large area data storage system according to claim *o 
6, wherein said first storage sub-system receives 
data from said second storage sub-system through 
synchronous transfer, and asynchronous transfer is 
used to transmit said data to said third storage sub- 
system. 45 

8. A large area data storage system according to claim 
6, wherein said first storage sub-system uses syn- 
chronous transfer to transmit data received from a 
host to said second storage sub-system and also so 
uses asynchronous transfer to transmit said data to 
said third storage sub-system. 

9. A large area data storage system according to claim 

6, wherein said first data center and said second 55 
data center are located in the vicinity, and said first 
data center and said third data center are situated 
at remote locations. 



10. A remote copy control method performed among 
three or more data centers, for a large data storage 
system (Fig. 1) which includes a first storage sub- 
system located in a first data center (1; Fig. 1), a 
second storage sub-system located in a second da- 
ta center (2) and a third sub-system located in a 
third data center (3), said remote copy control meth- 
od comprising: 

a first step of said first storage sub-system re- 
ceiving data from a host; 
a second step of said first storage sub-system 
synchronously transmitting, to said second 
storage sub-system, said data received from 
said host; 

a third step of said first storage sub-system 
asynchronously transmitting, to said third stor- 
age sub-system, said data received from said 
host; and 

a fourth step of said second storage sub-sys- 
tem transmitting a inquiry to said third storage 
sub-system to determine whether said data 
from said host has arrived at said third storage 
sub-system. 

11. A remote copy control method according to claim 
1 0, further comprising: 

a fifth step of halting the function of said first 
data center; and 

a sixth step of said second storage sub-system 
transmitting, to said third storage sub-system, 
a part of the data stored in said second storage 
sub-system. 

12. A remote copy control method, performed among 
three or more data centers, for a large data storage 
system (Fig. 1 ) that includes a first storage sub-sys- 
tem located in a first data center (1; Fig. 1), a second 
storage sub-system located in a second data center 
(2) and a third sub-system located in a third data 
center (3), said remote copy control method com- 
prising: 

a first step of said first storage sub-system re- 
ceiving data from a host; 
a second step of said first storage sub-system 
synchronously transmitting, to said second 
storage sub-system, said data received from 
said host; 

a third step of said first storage sub-system 
asynchronously transmitting, to said third stor- 
age sub-system, said data received from said 
host through synchronous transfer; and 
a fourth step of said first storage sub-system 
issuing an inquiry to said third storage sub-sys- 
tem to determine whether said data from said 
host has arrived at said third storage sub-sys- 
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tem. 

13. A remote copy control method according to claim 
12, further comprising: 

a fifth step of halting the function of said second 
data center; and 

a sixth step of said first storage sub-system 
transmitting, to said third storage sub-system, 
one part of data stored in said first storage sub- 
system. 

14. A remote copy control method, for a storage sub- 
system (1 ; Fig. 10) that includes means for writing 
data to a storage resource and that is connected to 
a plurality of transmission destinations to which da- 
ta stored in said storage resource is to be transmit- 
ted, comprising the steps of: 

said storage sub-system writing data to said 
storage resource; 

said storage sub-system storing a correlation 
(Fig. 13) between write position information, 
which specifies a position in said storage re- 
source whereat said data has been written, and 
a sequence number that is provided in the data 
writing order; 

said storage sub-system transmitting, to one of 
said transmission destinations, said data that 
has been written and said correlation (S123; 
Fig. 16); 

another storage sub-system (2; Fig. 16) for re- 
ceiving said sequence number from said trans- 
mission destination; and 
said storage sub-system employing a correla- 
tion stored therein and said received sequence 
number to obtain written data that has not yet 
been reflected to said transmission destination 
(S126; Fig. 16). 

15. 15. A remote copy control method, for a large 
data storage system including a first storage 
sub-system that has means for writing data to 
a first storage resource and that is located in a 
first data center (1 ; Fig. 1), a host connected to 
said first storage sub-system, a second storage 
sub-system (2; Fig. 1) that includes means for 
writing data to a second storage resource and 
that is located in a second data center, and a 
third sub-system (3; Fig. 1) that includes means 
for writing data to a third storage resource and 
that is located in a third data center, said remote 
copy control method comprising the steps of: 

said first storage sub-system writing data 
to said first storage resource in accordance 
with an instruction from said host; 
said first storage sub-system storing a cor- 



relation (Fig. 13) between write position in- 
formation, which specifies a position in said 
first storage resource whereat said data 
has been written, and a sequence number 

5 that is provided in the data writing order; 

said first storage sub-system transmitting, 
to said second storage sub-system, said 
data that has been written and said corre- 
lation (S123; Fig. 16); 

10 said second storage sub-system storing, in 

said second storage resource, said data 
and said correlation that have been re- 
ceived, and transmitting said data and said 
correlation to said third storage sub-sys- 

15 tem (S1 24; Fig. 16); 

said third storage sub-system receiving 
said data and said correlation, storing said 
data in said third storage resource and 
transmitting, to said first storage sub-sys- 

20 tem, said sequence number in accordance 

with said correlation (S126; Fig. 16); and 
said first storage sub-system employing 
said received sequence and a correlation 
stored therein to obtain data that has not 

25 yet been reflected to said third storage re- 

source (S126; Fig. 16). 

16. A remote copy control method according to claim 
1 5, further comprising the steps of: 

30 

when said second storage sub-system breaks 
down due to a blockage, said first storage sub- 
system transmitting to said third storage sub- 
system differential data that has been obtained 

35 based on said received sequence number and 

said stored correlation and that has not yet 
been reflected to said third storage resource, 
and write position information (S132; Fig. 17) 
for said differential data (S133; Fig. 17); 

40 said third storage sub-system receiving said 

differential data and said write position informa- 
tion, and storing said differential data in said 
third storage resource based on said write po- 
sition information (S133; Fig. 17) in order to 

45 synchronize the contents of said first storage 

resource and the contents of said third storage 
resource; 

when data is written to said first storage re- 
source in accordance with an instruction from 

50 said host, said first storage sub-system trans- 

mitting, to said third storage sub-system, said 
written data and write position information that 
specifies a position in said first storage re- 
source whereat said data has been written 

55 (S134; Fig. 17); and 

said third storage sub-system receiving said 
data and said write position information, and 
storing said data at a location in said third stor- 
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age resource that is specified by said write po- 
sition information. 

17. A remote copy control method according to claim 
16, further comprising the steps of: 

when said second storage sub-system has 
been recovered and is again available, said first 
storage sub-system transmitting all the data 
stored in said first storage resource to said sec- 
ond storage resource; 

said first storage sub-system writing data to 
said first storage resource in accordance with 
an instruction from said host, and transmitting 
said data and said sequence number to said 
second and third storage sub-systems; 
said second storage sub-system receiving said 
data and said sequence number, and storing 
said data in said second storage resource; 
said second storage sub-system storing a cor- 
relation between write position information, 
which specifies a location in said second stor- 
age resource whereat said data has been writ- 
ten, and said sequence numberthat is provided 
in the data writing order; 
said third storage sub-system receiving said 
data and said sequence number from said first 
storage sub-system, storing said data in said 
third storage resource, and transmitting to said 
second storage sub-system said sequence 
number according to said correlation; 
halting data transmission by said first storage 
sub-system to said second storage sub-sys- 
tem; 

said second storage sub-system receiving said 
sequence number from said third storage sub- 
system, and employing said sequence number, 
said sequence number stored therein and cor- 
responding write position information to obtain 
data that has not yet been reflected to said third 
storage resource; 

said second storage sub-system transmitting to 
said third storage sub-system the obtained dif- 
ferential data that has not been reflected to said 
third storage resource, and write position infor- 
mation; and 

said third storage sub-system receiving said 
differential data and said write position informa- 
tion, and storing said differential data in said 
third storage resource based on said write po- 
sition information so as to obtain synchroniza- 
tion between the contents of said first storage 
resource and the contents of said third storage 
resource. 

18. A remote copy control method, for a storage sub- 
system (1 ; Fig. 10) which includes means for writing 
data to a storage resource and which is connected 



to a plurality of transmission destinations to which 
data stored in said storage resource is to be trans- 
mitted, comprising the steps of: 

5 said storage sub-system writing data to said 

storage resource; 

said storage sub-system generating a correla- 
tion (Fig. 13) between write position informa- 
tion, which specifies a position in said storage 
w resource whereat said data has been written, 

and a sequence numberthat is provided in the 
data writing order; 

said storage sub-system transmitting to a des- 
tination A, which is one of said transmission 

15 destinations, said data that has been written 

and said correlation (S162; Fig. 20); 
said destination A storing said correlation re- 
ceived from said storage sub-system; 
said destination A receiving a sequence 

20 number from a destination B that is another one 

of said transmission destinations; 
another storage sub-system (2; Fig. 1 6) receiv- 
ing said sequence number from said transmis- 
sion destination; and 

25 said destination A employing a correlation 

stored therein and said sequence number re- 
ceived from said destination B to obtain written 
data that has not yet been reflected to said 
transmission destination (S166; Fig. 20). 

30 

1 9. A remote copy control method , for a large data stor- 
age system including a first storage sub-system that 
includes means for writing data to a first storage re- 
source and that is located in a first data center (1 ; 

35 Fig. 1 ), a second storage sub-system (2; Fig. 1 ) that 
includes means for writing data to a second storage 
resource and that is located in a second data center, 
a host connected to said second storage sub-sys- 
tem, and a third sub-system (3; Fig. 1)that includes 

40 means for writing data to a third storage resource 
and that is located in a third data center, said remote 
copy control method comprising the steps of: 

said second storage sub-system writing data to 
45 said second storage resource in accordance 

with an instruction from said host (S161; Fig. 
20); 

said second storage sub-system transmitting, 
to said first storage sub-system, said data that 

so has been written, write position information that 

specifies a position whereat said data has been 
written, and a sequence numberthat is provid- 
ed in the data writing order (S162; Fig. 20); 
said first storage sub-system storing in said first 

55 storage resource said data and said sequence 

number that have been received, and storing 
said sequence number in correlation with said 
write position information that specifies a stor- 
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age position in said first storage resource 
whereat said data has been written (S163; Fig. 
20); 

said second storage sub-system transmitting 
said data and said sequence number to said 
third storage sub-system (S164; Fig. 20); 
said third storage sub-system receiving said 
data and said sequence number, storing said 
data in said third storage resource, and trans- 
mitting, to said first storage sub-system, said 
sequence number that is paired with said data 
(S165; Fig. 20); and 

said first storage sub-system employing said 
received sequence and a correlation stored 
therein to obtain data that has not yet been re- 
flected to said third storage resource (S166; 
Fig. 20). 

20. A remote copy controi method according to claim 

19, further comprising the steps of: 

when said second storage sub-system breaks 
down due to a blockage, said first storage sub- 
system transmitting, to said third storage sub- 
system, differential data that has been obtained 
based on said received sequence number and 
said stored correlation and that has not yet 
been reflected to said third storage resource, 
and write position information (S172; Fig. 21) 
for said differential data (S173; Fig. 21); 
said third storage sub-system receiving said 
differential data and said write position informa- 
tion, and storing said differentia] data in said 
third storage resource, based on said write po- 
sition information (S173; Fig. 21), to synchro- 
nize the contents of said first storage resource 
and the contents of said third storage resource; 
when data is written to said first storage re- 
source in accordance with an instruction from 
said host, said first storage sub-system trans- 
mitting, to said third storage sub-system, said 
written data and write position information that 
specifies a location in said first storage re- 
source whereat said data has been written 
(S174; Fig. 21); and 

said third storage sub-system receiving said 
data and said write position information, and 
storing said data at a position in said third stor- 
age resource that is specified by said write po- 
sition information. 

21 . A remote copy control method according to claim 

20, further comprising the steps of: 

when said second storage sub-system is recov- 
ered and is again available, said first storage 
sub-system transmitting to said second storage 
resource all the data that has been written by 



said host and is stored in said first storage re- 
source; 

said first storage sub-system writing data to 

said first storage resource in accordance with 

an instruction from said host; 

said first storage sub-system transmitting said 

data and said sequence number to said second 

and third storage sub-systems; 

said second storage sub-system receiving said 

data and said sequence number, and storing 

said data in said second storage resource at an 

appropriate time; 

said second storage sub-system storing a cor- 
relation between write position information, 
which specifies a position in said second stor- 
age resource whereat said data has been writ- 
ten, and said sequence number that is provided 
in the data writing order; 
said third storage sub-system receiving said 
data and said sequence number, storing said 
data in said third storage resource, and trans- 
mitting to said second storage sub-system said 
sequence number according to said correla- 
tion; 

halting data transmission by said first storage 
sub-system to said second storage sub-sys- 
tem; 

said second storage sub-system receiving said 
sequence number, and employing said se- 
quence number and a correlation stored therein 
to obtain data that has not yet been reflected to 
said third storage resource; 
said second storage sub-system transmitting to 
said third storage sub-system differential data 
that has been obtained based on said se- 
quence number and said correlation and that 
has not yet been reflected to said third storage 
resource, and write position information for said 
differential data; and 

said third storage sub-system receiving said 
differential data and said write position informa- 
tion, and storing said differential data in said 
third storage resource based on said write po- 
sition information to synchronize the contents 
of said first storage resource and the contents 
of said third storage resource. 

22. A remote copy control method according to one of 
claims 15 to 17 and 19 to 21 , wherein synchronous 
transfer is employed for the operation of said first 
storage sub-system and said second storage sub- 
system, and wherein asynchronous transfer is em- 
ployed for the operation of said first storage sub- 
system and said third storage sub-system. 

23. A remote copy control method according to one of 
claims 15 to 21 , wherein said second storage sub- 
system or said third storage sub-system stores data 
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at a storage location in said second storage re- 
source or said third storage resource that corre- 
sponds to said write position information stored in 
said first storage sub-system. 

24. A remote copy control method, for a large area data 
storage system having a storage sub-system A(1) 
connected to a host and a plurality of other storage 
sub-systems B(2, 3) connected to said storage sub- 
system through communication means, comprising 
the steps of: 

said storage sub-systems B detecting a block- 
age that has occurred at one, at the least, of 
said host, said storage sub-system A and said 
communication means; 
when said storage sub-systems B have detect- 
ed said blockage, selecting a storage sub-sys- 
tem b(3) from among said storage sub-systems 
B as a proxy for preforming the process for said 
storage sub-system A; 

matching the data contents of said selected 
storage sub-system b and the data contents of 
other storage sub-systems B; and 
said storage sub-systems transferring the op- 
eration of said host from a sub-host that is con- 
nected to said storage sub-system b. 

25. A remote copy control method according to claim 
24, wherein one or more of said storage sub-sys- 
tems B perform said step of: 

detecting the occurrence of said blockage, and 
said step of selecting said storage sub-system 
b as a proxy for performing the process of said 
storage sub-system A. 

26. A remote copy control method according to claim 
24, wherein said step of detecting the occurrence 
of said blockage is a step of: 

identifying that a blockage has occurred when 
a heart beat message from said storage sub- 
system A is not received at a predetermined 
time by said storage sub-systems B. 

27. A remote copy control method according to claim 
24, wherein said step, upon detecting said block- 
age, of selecting said storage sub-system b from 
among said storage sub-systems B as a proxy for 
performing the process of said storage sub-system 
A is a step of: 

comparing the latest sequence numbers 
among the sequence numbers that represent 
the order for the updating of data (Fig. 15) 
stored in said storage sub-systems B, and se- 
lecting a storage sub-system B that stores the 



latest sequence number as said storage sub- 
system b that serves as a proxy for performing 
the process of said storage sub-system A. 

5 28. A remote copy control method according to claim 
24, wherein, when of said sequence numbers indi- 
cating the order for the updating of data stored in 
each of said storage sub-systems B one sequence 
number is omitted, the latest of the sequential se- 

10 quence numbers is employed for the comparison. 

29. A remote copy control method according to claim 
24, wherein said step of matching the data contents 
of said selected storage sub-system b and the data 
15 contents of the other storage sub-systems B is a 
step of: 

copying all the data between said storage sub- 
system b and the other storage sub-systems B, 
20 or copying differential data stored in said stor- 

age sub-systems, so that the contents of said 
storage sub-system b match the contents of 
said other storage sub-systems B. 

25 30. A storage sub-system, which includes means for 
writing data to a storage resource, which is connect- 
ed to a plurality of transmission destinations to 
which data stored in said storage resource is to be 
transmitted and which is used for a remote copy 

30 control method according to claim 14, comprising: 

means for writing data to said storage resource; 
means for storing a correlation (Fig. 13) be- 
tween write position information, which speci- 
35 fjes a position in said storage resource whereat 

said data has been written, and a sequence 
number that is provided in the data writing or- 
der; 

means for transmitting, to one of said transmis- 
40 sion destinations, said data that has been writ- 

ten and said correlation; 
means for receiving said sequence number 
from said transmission destination; and 
means for employing a correlation stored there- 
45 in and said received sequence number to ob- 

tain written data that has not yet been reflected 
to said transmission destination. 

31 . A storage sub-system for a large data storage sys- 
50 tern including a first storage sub-system that in- 
cludes means for writing data to a first storage re- 
source and that is located in a first data center (1 ; 
Fig. 1 ), a host connected to said first storage sub- 
system, a second storage sub-system (2; Fig. 1) 
55 that includes means for writing data to a second 
storage resource and that is located in a second da- 
ta center, and a third sub-system (3; Fig. 1) that in- 
cludes means for writing data to a third storage re- 
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source and that is located in a third data center, said 
storage sub-system that serves as said first storage 
sub-system used for a remote copy control method 
according to claim 15 comprising: 

means for writing data to said first storage re- 
source in accordance with an instruction from 
said host; 

means for storing a correlation (Fig. 13) be- 
tween write position information, which speci- 
fies a position in said first storage resource 
whereat said data has been written, and a se- 
quence number that is provided in the data writ- 
ing order; 

means for transmitting, to said second storage 
sub-system, said data that has been written 
and said correlation; 

means for storing, in said second storage re- 
source, said data and said correlation that have 
been received, and transmitting said data and 
said correlation to said third storage sub-sys- 
tem; 

means for receiving said data and said correla- 
tion, storing said data in said third storage re- 
source and transmitting, to said first storage 
sub-system, said sequence number in accord- 
ance with said correlation; and 
means for employing said received sequence 
and a correlation stored therein to obtain data 
that has not yet been reflected to said third stor- 
age resource. 

A storage sub-system for a storage sub-system A 
that includes means for writing data to a storage re- 
source and that is connected to a plurality of trans- 
mission destinations to which data stored in said 
storage resource is to be transmitted, said storage 
sub-system that serves as one of said transmission 
destinations used for a remote copy control method 
according to claim 18 comprising: 

means for storing data that is received from 
said storage sub-system A and is written to said 
storage resource, and a correlation (Fig. 13) 
between write position information, which spec- 
ifies a position in said storage resource whereat 
said data has been written, and a sequence 
number that is provided in the data writing or- 
der; 

means for receiving a sequence number from 
a destination B that is another one of said trans- 
mission destinations; 

means for employing a correlation stored there- 
in and said sequence number received from 
said destination B to obtain written data that 
has not yet been reflected to said transmission 
destination. 
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33. A storage sub-system for a large data storage sys- 
tem including a first storage sub-system that in- 
cludes means for writing data to a first storage re- 
source and that is located in a first data center, a 
second storage sub-system that includes means for 
writing data to a second storage resource and that 
is located in a second data center, a host connected 
to said second storage sub-system, and a third sub- 
system that includes means for writing data to a 
third storage resource and that is located in a third 
data center, said storage sub-system that serves as 
said first storage sub-system used for a remote 
copy control method according to claim 1 9 compris- 
ing: 

means for receiving, from said second storage 
sub-system, data that is written in accordance 
with an instruction from said host, and a corre- 
lation between write position information, which 
specifies a position in said storage resource 
whereat said data has been written, and a se- 
quence number that is provided in the data writ- 
ing order, and for storing said data and said cor- 
relation in said first storage resource; 
means for receiving a sequence number from 
said third storage sub-system, and for employ- 
ing said sequence number and said correlation 
stored therein to obtain written data that has not 
yet been reflected to said third storage sub-sys- 
tem. 



34. A storage sub-system for a large area data storage 
system having a storage sub-system A connected 
to a host and a plurality of other storage sub-sys- 
35 terns B connected to said storage sub-system 
through communication means, said storage sub- 
system that serves as said storage sub-system B 
used for a remote copy control method according 
to claim 24 comprising: 

40 

means for detecting a blockage that has oc- 
curred at one, at the least, of said host, said 
storage sub-system A and said communication 
means; 

45 means for, when said storage sub-systems B 

have detected said blockage, selecting a stor- 
age sub-system b from among said storage 
sub-systems B as a proxy for preforming the 
process for said storage sub-system A; 

so means for matching the data contents of said 

selected storage sub-system b and the data 
contents of other storage sub-systems B; and 
means for transferring the operation of said 
host from a sub-host that is connected to said 

55 storage sub-system b. 
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FIG. 11 
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FIG. 20A 
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